Hi all,

I think that SUnit's traffic light would be more reliable if we (1) introduce #skippedTests (being as flexible as #expectedFailures), (2) migrate all expected failures to that new form of skipped tests, and (3) deprecate the idea of expected failures in SUnit. That is, #skippedTests *should not run* because on certain platforms, they don't make sense. In all other cases, tests are just tests and are expected to pass. Consequently, there would be no arguing about how to handle "unexpected passes" in tools anymore. The traffic light would be fine.

Best,
Marcel
Am 22.06.2019 00:24:45 schrieb Chris Muller <asqueaker at gmail.com>:
Hi Bernhard,

> Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.

The three-state Status Indicator ( (green | yellow | red) = (okay |
warning | error) ) is one of the most widely-studied and important
outputs of many applications. Unfortunately for SUnit, having tests
that are known not to pass on certain platforms basically destroys
that Status output.

> It is a pity that the trunk is not green all the time.

Until we fix our use of #expectedFailures, it'll never report green.
The primary requirement of SUnit is supposed to report the status of
the software, and relieve users from needing to dig into the details
just to ascertain that. In fact, that digging is actually 95% of the
work of running the tests oneself, the other 5% being the single click
launch of the Suite.

So, that external patch file specifying internal class names is a
desperate hack trying to make up for #expectedFailures not doing what
it's supposed to. We really should fix our SUnit.

> I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file.

+1. But this doesn't fix the quick output Status, so we should still
fix SUnit, too.

> It might help to increase the urge to fix them.

I think we will always have some #expectedFailures, though. It seems
to be a useful and necessary feature (except for how it subverts the
green Status, of course).

> Besides it would make it easier to find out whether one has introduced a regression.

Exactly.

Although Kent Beck's original SUnit article [1] didn't anticipate the
need for an #expectedFailures mechanism, he did, under "Philosophy",
clearly define a "failure":

"A failure is an anticipated problem. When you write tests,
you check for expected results. If you get a different answer, that is
a failure."

Additionally, check out the original 2003 implementation of
TestResult>>#passed still in trunk:

passed
^ self expectedPasses, self expectedDefects

There it is. "passed" includes the #expectedDefects, which are the
'errors' + 'failures' which are expected to fail, so this original
part of the implementation, too, is consistent with Beck's definition,
above. It also shows my own hack fix (SUnit-cmm.116.mcz in the Inbox)
is not a complete and/or proper fix, but it has the correct intention.

We definitely should find a consensus on a fix for this.

Best,
Chris

[1] - http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20190624/89d3f912/attachment.html>