[squeak-dev] The Inbox: SUnit-cmm.116.mcz

Marcel Taeumel marcel.taeumel at hpi.de
Wed May 29 07:15:54 UTC 2019

Hi Chris,

if an unexpected pass would be treated as a failure, *you would encourage* tests like this:

   self assert: 5 / 0.

   ^ #(testSomething)

A programmer's goal should always be to write passing tests. An expected failure should only be temporary. If that expected failure passes at some point without the programmer noticing and updating #expectedFailures, that's not a bad thing. It's progress. The programmer will update #expectedFailures eventually.

The list of expected failures is just an add-on to your test suite to document not-yet-working stuff so that fellow programmers do not treat those failing tests as some new regression. 

If you treat expected failures as something more inherent in your test suite, you would encourage tests like the one sketched above. I wouldn't do that.

Am 29.05.2019 00:42:50 schrieb Chris Muller <ma.chris.m at gmail.com>:
Hey Marcel,

I think we either disagree what #expectedFailures is for, or one of us forgot (possibly me).  When I "zoomed out" earlier in the thread to explain my real world use-case, it was an implicit invitation for alternative solutions.  Would you mind helping me out with one then?

Because I assume we agree on these fundamental requirements of SUnit..

  1) allow user to know the result of running tests via a gross glance from across the room (e.g., if I see "green", I know all is good).

  2) Be able to interrogate the result of running tests *programmatically* (e.g., "myResult failures isEmpty"...?).

  3) Be able to document functionality *expected* to fail (independent of "should"), while keeping the above requirements in line with these expectations.

..but by saying that expectedFailures is just "an FYI list off-to-the-side for documentation" -- and treating them the same as regular failures basically neuters all the above basic capabilities of SUnit.  We're forcing it to report false negatives *100% of the time*, which means it'll *never report green*.

> But expectedFailures are a subset *within* a domain-specific reason that

> not only *should* (IMO), but *need* to be separated out, because SUnit
> provides registration of expectedFailures ONLY at the #test... method
> level.

Hmm... no. Given that expected failures work at the level of a #test... method, programmers can just work with that. Actually, SUnit relies on the programmer to figure out the best granularity for both passing and failing tests. There is no need to do some analysis that goes sub-method here. 

Despite the "no", I actually agree with everything you said after that...  I just think that "figuring out the best granularity" involves doing "sub-method" analysis automatically, without even realizing it.

Your proposed change in TestResult >> #unexpectedPasses renders the code practically unreadable:

  ^ failures select: [:each | each shouldPass not]

What does it say?  Well, somehow, a "pass" got stored in the variable "failures" and if you happen to know that #shouldPass can sort out those hidden passes... way too much magic.

The fact you put "pass" in quotes, above, means, deep down, I think you agree with me, :)   that going against the programmers *declared expectations* is NOT a pass, which is why it's rightly found in the failures collection.

This was your initial fallacy for this proposal:

> The Result of a Test which is tagged as an #expectedFailure IS a failure if the test passed.

No, it is not. :-) If anything, the programmer could be happy that an expected failure passes. It is *not a failure*. Maybe something third, which we don't have yet. Serendipity? Happy accident? Not a failure.

You can't possibly believe that and articulate a rational reason for the existence of #expectedFailures.

LOL, and it wouldn't be "happy" because you wouldn't even know it happened!  The SUnit browser would just be the same yellow it always is, with a same-looking smattering of "expectedFailures" listed at the top, maybe even with one REAL failure mixed in, so that the size of the list looks just right, and so you won't even bother checking manually the actual testMethodNames which failed to make sure they're truly ALL the "expected" ones.  It could be until years later, when another programmer would find it sitting in "expectedFailures" and get confused why its in there.

Please help me to rediscover the capabilities of SUnit that made it actually useful.

 - Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20190529/3ba247fb/attachment.html>

More information about the Squeak-dev mailing list