Hey Marcel,

Consider future readers of the #test.. methods which are skipped in this
way -- all the code there with no conditions on its execution.  Then they
run the Suite and it reports all green, woo hoo!  Even if they knew about
#skippedTests going in, it's easy to imagine them forgetting to cross-check
their code reading against that list, so the #test.. methods they just read
in the browser were implicitly conveyed to them by the system as "passing"
when, in fact, they didn't even run.

SUnit derives its value by efficient and correct conveyance of the status
of the software.

I'm sure we'll figure something out.

Best,
  Chris


On Mon, Jun 24, 2019 at 1:12 AM Marcel Taeumel <marcel.taeumel at hpi.de>
wrote:

> Hi all,
>
> I think that SUnit's traffic light would be more reliable if we (1)
> introduce #skippedTests (being as flexible as #expectedFailures), (2)
> migrate all expected failures to that new form of skipped tests, and (3)
> deprecate the idea of expected failures in SUnit. That is, #skippedTests
> *should not run* because on certain platforms, they don't make sense. In
> all other cases, tests are just tests and are expected to pass.
> Consequently, there would be no arguing about how to handle "unexpected
> passes" in tools anymore. The traffic light would be fine.
>
> Best,
> Marcel
>
> Am 22.06.2019 00:24:45 schrieb Chris Muller <asqueaker at gmail.com>:
> Hi Bernhard,
>
> > Thanks for the clarification. I did not know there was a separate
> mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the
> image. Do you happen to know what is the rationale for that? Especially
> when the build succeeds anyway.
>
> The three-state Status Indicator ( (green | yellow | red) = (okay |
> warning | error) ) is one of the most widely-studied and important
> outputs of many applications. Unfortunately for SUnit, having tests
> that are known not to pass on certain platforms basically destroys
> that Status output.
>
> > It is a pity that the trunk is not green all the time.
>
> Until we fix our use of #expectedFailures, it'll never report green.
> The primary requirement of SUnit is supposed to report the status of
> the software, and relieve users from needing to dig into the details
> just to ascertain that. In fact, that digging is actually 95% of the
> work of running the tests oneself, the other 5% being the single click
> launch of the Suite.
>
> So, that external patch file specifying internal class names is a
> desperate hack trying to make up for #expectedFailures not doing what
> it's supposed to. We really should fix our SUnit.
>
> > I wonder how difficult it would be to include information on failing
> tests in the download packages say as a known_test_failures.txt alongside
> the image file.
>
> +1. But this doesn't fix the quick output Status, so we should still
> fix SUnit, too.
>
> > It might help to increase the urge to fix them.
>
> I think we will always have some #expectedFailures, though. It seems
> to be a useful and necessary feature (except for how it subverts the
> green Status, of course).
>
> > Besides it would make it easier to find out whether one has introduced a
> regression.
>
> Exactly.
>
> Although Kent Beck's original SUnit article [1] didn't anticipate the
> need for an #expectedFailures mechanism, he did, under "Philosophy",
> clearly define a "failure":
>
> "A failure is an anticipated problem. When you write tests,
> you check for expected results. If you get a different answer, that is
> a failure."
>
> Additionally, check out the original 2003 implementation of
> TestResult>>#passed still in trunk:
>
> passed
> ^ self expectedPasses, self expectedDefects
>
> There it is. "passed" includes the #expectedDefects, which are the
> 'errors' + 'failures' which are expected to fail, so this original
> part of the implementation, too, is consistent with Beck's definition,
> above. It also shows my own hack fix (SUnit-cmm.116.mcz in the Inbox)
> is not a complete and/or proper fix, but it has the correct intention.
>
> We definitely should find a consensus on a fix for this.
>
> Best,
> Chris
>
> [1] -
> http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20190624/722bffb1/attachment.html>