On Wed., May 29, 2019, 00:16 Marcel Taeumel, <marcel.taeumel at hpi.de> wrote:

> Hi Chris,
>
> if an unexpected pass would be treated as a failure, *you would encourage*
> tests like this:
>
> testSomething
>    self assert: 5 / 0.
>
> expectedFailures
>    ^ #(testSomething)
>
> A programmer's goal should always be to write passing tests. An expected
> failure should only be temporary. If that expected failure passes at some
> point without the programmer noticing and updating #expectedFailures,
> that's not a bad thing. It's progress. The programmer will update
> #expectedFailures eventually.
>
> The list of expected failures is just an add-on to your test suite to
> document not-yet-working stuff so that fellow programmers do not treat
> those failing tests as some new regression.
>
> If you treat expected failures as something more inherent in your test
> suite, you would encourage tests like the one sketched above. I wouldn't do
> that.
>
> Best,
> Marcel
>

But where is the incentive for a programmer to fix a test and remember to
update #expectedFailures?

On the other hand there might be tests that flake, but no one has gotten
around to deflaking it and we suppress the test with #expectedFailures, when
it passes 80% of the time we want to not fail builds 20% of the time. (Ok
really we do, but not many of us are paid to keep the build green.)

frank

> Am 29.05.2019 00:42:50 schrieb Chris Muller <ma.chris.m at gmail.com>:
> Hey Marcel,
>
> I think we either disagree what #expectedFailures is for, or one of us
> forgot (possibly me).  When I "zoomed out" earlier in the thread to explain
> my real world use-case, it was an implicit invitation for alternative
> solutions.  Would you mind helping me out with one then?
>
> Because I assume we agree on these fundamental requirements of SUnit..
>
>   1) allow user to know the result of running tests via a gross glance
> from across the room (e.g., if I see "green", I know all is good).
>   2) Be able to interrogate the result of running tests *programmatically*
> (e.g., "myResult failures isEmpty"...?).
>   3) Be able to document functionality *expected* to fail (independent of
> "should"), while keeping the above requirements in line with these
> expectations.
>
> ..but by saying that expectedFailures is just "an FYI list off-to-the-side
> for documentation" -- and treating them the same as regular failures
> basically neuters all the above basic capabilities of SUnit.  We're forcing
> it to report false negatives *100% of the time*, which means it'll *never
> report green*.
>
> > But expectedFailures are a subset *within* a domain-specific reason that
>>
> > not only *should* (IMO), but *need* to be separated out, because SUnit
>> > provides registration of expectedFailures ONLY at the #test... method
>> > level.
>>
>> Hmm... no. Given that expected failures work at the level of a #test...
>> method, programmers can just work with that. Actually, SUnit relies on the
>> programmer to figure out the best granularity for both passing and failing
>> tests. There is no need to do some analysis that goes sub-method here.
>>
>
> Despite the "no", I actually agree with everything you said after that...
> I just think that "figuring out the best granularity" involves doing
> "sub-method" analysis automatically, without even realizing it.
>
>
>> Your proposed change in TestResult >> #unexpectedPasses renders the code
>> practically unreadable:
>>
>> unexpectedPasses
>>   ^ failures select: [:each | each shouldPass not]
>>
>> What does it say?  Well, somehow, a "pass" got stored in the variable
>> "failures" and if you happen to know that #shouldPass can sort out those
>> hidden passes... way too much magic.
>>
>
> The fact you put "pass" in quotes, above, means, deep down, I think you
> agree with me, :)   that going against the programmers *declared
> expectations* is NOT a pass, which is why it's rightly found in the
> failures collection.
>
>
>> This was your initial fallacy for this proposal:
>>
>> > The Result of a Test which is tagged as an #expectedFailure IS a
>> failure if the test passed.
>>
>> No, it is not. :-) If anything, the programmer could be happy that an
>> expected failure passes. It is *not a failure*. Maybe something third,
>> which we don't have yet. Serendipity? Happy accident? Not a failure.
>>
>
> You can't possibly believe that and articulate a rational reason for the
> existence of #expectedFailures.
>
> LOL, and it wouldn't be "happy" because you wouldn't even know it
> happened!  The SUnit browser would just be the same yellow it always is,
> with a same-looking smattering of "expectedFailures" listed at the top,
> maybe even with one REAL failure mixed in, so that the size of the list
> looks just right, and so you won't even bother checking manually the actual
> testMethodNames which failed to make sure they're truly ALL the "expected"
> ones.  It could be until years later, when another programmer would find it
> sitting in "expectedFailures" and get confused why its in there.
>
> Please help me to rediscover the capabilities of SUnit that made it
> actually useful.
>
>  - Chris
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20190529/2fa3a96f/attachment.html>