It seems that the concept of expected failures is hard to grasp. Other communities seem to struggle as well. See StackOverflow. A skip list might be better, maybe via pragma in the test itself. Not sure. Looking at all those #expectedFailures in the image, there are many not-so-temporary uses of it, but platform-specific conditions etc. Oh, well...

- auto skip list that documents platform-specific dependencies? (unix, network, ...)
- manual skip list that indicates work-in-progress tests?


At least this would avoid the discussion about unexpected passes ... sigh...

Best,
Marcel



Am 29.05.2019 18:34:29 schrieb Frank Shearar <frank.shearar at gmail.com>:

On Wed., May 29, 2019, 00:16 Marcel Taeumel, <marcel.taeumel at hpi.de [mailto:marcel.taeumel at hpi.de]> wrote:

Hi Chris,

if an unexpected pass would be treated as a failure, *you would encourage* tests like this:

testSomething
   self assert: 5 / 0.

expectedFailures
   ^ #(testSomething)

A programmer's goal should always be to write passing tests. An expected failure should only be temporary. If that expected failure passes at some point without the programmer noticing and updating #expectedFailures, that's not a bad thing. It's progress. The programmer will update #expectedFailures eventually.

The list of expected failures is just an add-on to your test suite to document not-yet-working stuff so that fellow programmers do not treat those failing tests as some new regression. 

If you treat expected failures as something more inherent in your test suite, you would encourage tests like the one sketched above. I wouldn't do that.

Best,
Marcel

But where is the incentive for a programmer to fix a test and remember to update #expectedFailures?

On the other hand there might be tests that flake, but no one has gotten around to deflaking it and we suppress the test with #expectedFailures, when it passes 80% of the time we want to not fail builds 20% of the time. (Ok really we do, but not many of us are paid to keep the build green.)

frank
Am 29.05.2019 00:42:50 schrieb Chris Muller <ma.chris.m at gmail.com [mailto:ma.chris.m at gmail.com]>:
Hey Marcel,

I think we either disagree what #expectedFailures is for, or one of us forgot (possibly me).  When I "zoomed out" earlier in the thread to explain my real world use-case, it was an implicit invitation for alternative solutions.  Would you mind helping me out with one then?

Because I assume we agree on these fundamental requirements of SUnit..

  1) allow user to know the result of running tests via a gross glance from across the room (e.g., if I see "green", I know all is good).

  2) Be able to interrogate the result of running tests *programmatically* (e.g., "myResult failures isEmpty"...?).

  3) Be able to document functionality *expected* to fail (independent of "should"), while keeping the above requirements in line with these expectations.

..but by saying that expectedFailures is just "an FYI list off-to-the-side for documentation" -- and treating them the same as regular failures basically neuters all the above basic capabilities of SUnit.  We're forcing it to report false negatives *100% of the time*, which means it'll *never report green*.

> But expectedFailures are a subset *within* a domain-specific reason that

> not only *should* (IMO), but *need* to be separated out, because SUnit
> provides registration of expectedFailures ONLY at the #test... method
> level.


Hmm... no. Given that expected failures work at the level of a #test... method, programmers can just work with that. Actually, SUnit relies on the programmer to figure out the best granularity for both passing and failing tests. There is no need to do some analysis that goes sub-method here. 

Despite the "no", I actually agree with everything you said after that...  I just think that "figuring out the best granularity" involves doing "sub-method" analysis automatically, without even realizing it.
 

Your proposed change in TestResult >> #unexpectedPasses renders the code practically unreadable:


unexpectedPasses
  ^ failures select: [:each | each shouldPass not]

What does it say?  Well, somehow, a "pass" got stored in the variable "failures" and if you happen to know that #shouldPass can sort out those hidden passes... way too much magic.

The fact you put "pass" in quotes, above, means, deep down, I think you agree with me, :)   that going against the programmers *declared expectations* is NOT a pass, which is why it's rightly found in the failures collection.


This was your initial fallacy for this proposal:

> The Result of a Test which is tagged as an #expectedFailure IS a failure if the test passed.

No, it is not. :-) If anything, the programmer could be happy that an expected failure passes. It is *not a failure*. Maybe something third, which we don't have yet. Serendipity? Happy accident? Not a failure.

You can't possibly believe that and articulate a rational reason for the existence of #expectedFailures.

LOL, and it wouldn't be "happy" because you wouldn't even know it happened!  The SUnit browser would just be the same yellow it always is, with a same-looking smattering of "expectedFailures" listed at the top, maybe even with one REAL failure mixed in, so that the size of the list looks just right, and so you won't even bother checking manually the actual testMethodNames which failed to make sure they're truly ALL the "expected" ones.  It could be until years later, when another programmer would find it sitting in "expectedFailures" and get confused why its in there.

Please help me to rediscover the capabilities of SUnit that made it actually useful.

 - Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20190529/7ed2b4cd/attachment.html>