Hey Marcel,

I think we either disagree what #expectedFailures is for, or one of us
forgot (possibly me).  When I "zoomed out" earlier in the thread to explain
my real world use-case, it was an implicit invitation for alternative
solutions.  Would you mind helping me out with one then?

Because I assume we agree on these fundamental requirements of SUnit..

  1) allow user to know the result of running tests via a gross glance from
across the room (e.g., if I see "green", I know all is good).
  2) Be able to interrogate the result of running tests *programmatically*
(e.g., "myResult failures isEmpty"...?).
  3) Be able to document functionality *expected* to fail (independent of
"should"), while keeping the above requirements in line with these
expectations.

..but by saying that expectedFailures is just "an FYI list off-to-the-side
for documentation" -- and treating them the same as regular failures
basically neuters all the above basic capabilities of SUnit.  We're forcing
it to report false negatives *100% of the time*, which means it'll *never
report green*.

> But expectedFailures are a subset *within* a domain-specific reason that
>
> not only *should* (IMO), but *need* to be separated out, because SUnit
> > provides registration of expectedFailures ONLY at the #test... method
> > level.
>
> Hmm... no. Given that expected failures work at the level of a #test...
> method, programmers can just work with that. Actually, SUnit relies on the
> programmer to figure out the best granularity for both passing and failing
> tests. There is no need to do some analysis that goes sub-method here.
>

Despite the "no", I actually agree with everything you said after that...
I just think that "figuring out the best granularity" involves doing
"sub-method" analysis automatically, without even realizing it.


> Your proposed change in TestResult >> #unexpectedPasses renders the code
> practically unreadable:
>
> unexpectedPasses
>   ^ failures select: [:each | each shouldPass not]
>
> What does it say?  Well, somehow, a "pass" got stored in the variable
> "failures" and if you happen to know that #shouldPass can sort out those
> hidden passes... way too much magic.
>

The fact you put "pass" in quotes, above, means, deep down, I think you
agree with me, :)   that going against the programmers *declared
expectations* is NOT a pass, which is why it's rightly found in the
failures collection.


> This was your initial fallacy for this proposal:
>
> > The Result of a Test which is tagged as an #expectedFailure IS a
> failure if the test passed.
>
> No, it is not. :-) If anything, the programmer could be happy that an
> expected failure passes. It is *not a failure*. Maybe something third,
> which we don't have yet. Serendipity? Happy accident? Not a failure.
>

You can't possibly believe that and articulate a rational reason for the
existence of #expectedFailures.

LOL, and it wouldn't be "happy" because you wouldn't even know it
happened!  The SUnit browser would just be the same yellow it always is,
with a same-looking smattering of "expectedFailures" listed at the top,
maybe even with one REAL failure mixed in, so that the size of the list
looks just right, and so you won't even bother checking manually the actual
testMethodNames which failed to make sure they're truly ALL the "expected"
ones.  It could be until years later, when another programmer would find it
sitting in "expectedFailures" and get confused why its in there.

Please help me to rediscover the capabilities of SUnit that made it
actually useful.

 - Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20190528/b67d7875/attachment.html>