Hi Chris,

so are you looking for information more close to an individaul test method or test result? I see two complementary approaches here: a) re-design SUnit test representations (i.e., the GUI) and b) re-design SUnit test specifications (i.e., the code).

Best,
Marcel
Am 25.06.2019 03:30:14 schrieb Chris Muller <asqueaker at gmail.com>:
Hey Marcel,

Consider future readers of the #test.. methods which are skipped in this way -- all the code there with no conditions on its execution.  Then they run the Suite and it reports all green, woo hoo!  Even if they knew about #skippedTests going in, it's easy to imagine them forgetting to cross-check their code reading against that list, so the #test.. methods they just read in the browser were implicitly conveyed to them by the system as "passing" when, in fact, they didn't even run.

SUnit derives its value by efficient and correct conveyance of the status of the software.

I'm sure we'll figure something out.


Best,

  Chris



On Mon, Jun 24, 2019 at 1:12 AM Marcel Taeumel <marcel.taeumel at hpi.de [mailto:marcel.taeumel at hpi.de]> wrote:

Hi all,

I think that SUnit's traffic light would be more reliable if we (1) introduce #skippedTests (being as flexible as #expectedFailures), (2) migrate all expected failures to that new form of skipped tests, and (3) deprecate the idea of expected failures in SUnit. That is, #skippedTests *should not run* because on certain platforms, they don't make sense. In all other cases, tests are just tests and are expected to pass. Consequently, there would be no arguing about how to handle "unexpected passes" in tools anymore. The traffic light would be fine.

Best,
Marcel
Am 22.06.2019 00:24:45 schrieb Chris Muller <asqueaker at gmail.com [mailto:asqueaker at gmail.com]>:
Hi Bernhard,

> Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.

The three-state Status Indicator ( (green | yellow | red) = (okay |
warning | error) ) is one of the most widely-studied and important
outputs of many applications. Unfortunately for SUnit, having tests
that are known not to pass on certain platforms basically destroys
that Status output.

> It is a pity that the trunk is not green all the time.

Until we fix our use of #expectedFailures, it'll never report green.
The primary requirement of SUnit is supposed to report the status of
the software, and relieve users from needing to dig into the details
just to ascertain that. In fact, that digging is actually 95% of the
work of running the tests oneself, the other 5% being the single click
launch of the Suite.

So, that external patch file specifying internal class names is a
desperate hack trying to make up for #expectedFailures not doing what
it's supposed to. We really should fix our SUnit.

> I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file.

+1. But this doesn't fix the quick output Status, so we should still
fix SUnit, too.

> It might help to increase the urge to fix them.

I think we will always have some #expectedFailures, though. It seems
to be a useful and necessary feature (except for how it subverts the
green Status, of course).

> Besides it would make it easier to find out whether one has introduced a regression.

Exactly.

Although Kent Beck's original SUnit article [1] didn't anticipate the
need for an #expectedFailures mechanism, he did, under "Philosophy",
clearly define a "failure":

"A failure is an anticipated problem. When you write tests,
you check for expected results. If you get a different answer, that is
a failure."

Additionally, check out the original 2003 implementation of
TestResult>>#passed still in trunk:

passed
^ self expectedPasses, self expectedDefects

There it is. "passed" includes the #expectedDefects, which are the
'errors' + 'failures' which are expected to fail, so this original
part of the implementation, too, is consistent with Beck's definition,
above. It also shows my own hack fix (SUnit-cmm.116.mcz in the Inbox)
is not a complete and/or proper fix, but it has the correct intention.

We definitely should find a consensus on a fix for this.

Best,
Chris

[1] - http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm [http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm]


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20190625/a8262c2d/attachment.html>