[squeak-dev] SqueakSSLTest failure/errors on macOS?

Chris Muller asqueaker at gmail.com
Fri Jun 21 22:23:58 UTC 2019


Hi Bernhard,

> Thanks for the clarification. I did not know there was a separate mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the image. Do you happen to know what is the rationale for that? Especially when the build succeeds anyway.

The three-state Status Indicator ( (green | yellow | red) = (okay |
warning | error) ) is one of the most widely-studied and important
outputs of many applications.  Unfortunately for SUnit, having tests
that are known not to pass on certain platforms basically destroys
that Status output.

> It is a pity that the trunk is not green all the time.

Until we fix our use of #expectedFailures, it'll never report green.
The primary requirement of SUnit is supposed to report the status of
the software, and relieve users from needing to dig into the details
just to ascertain that.  In fact, that digging is actually 95% of the
work of running the tests oneself, the other 5% being the single click
launch of the Suite.

So, that external patch file specifying internal class names is a
desperate hack trying to make up for #expectedFailures not doing what
it's supposed to.  We really should fix our SUnit.

> I wonder how difficult it would be to include information on failing tests in the download packages say as a known_test_failures.txt alongside the image file.

+1.  But this doesn't fix the quick output Status, so we should still
fix SUnit, too.

> It might help to increase the urge to fix them.

I think we will always have some #expectedFailures, though.  It seems
to be a useful and necessary feature (except for how it subverts the
green Status, of course).

> Besides it would make it easier to find out whether one has introduced a regression.

Exactly.

Although Kent Beck's original SUnit article [1] didn't anticipate the
need for an #expectedFailures mechanism, he did, under "Philosophy",
clearly define a "failure":

         "A failure is an anticipated problem. When you write tests,
you check for expected results. If you get a different answer, that is
a failure."

Additionally, check out the original 2003 implementation of
TestResult>>#passed still in trunk:

        passed
              ^ self expectedPasses, self expectedDefects

There it is.  "passed" includes the #expectedDefects, which are the
'errors' + 'failures' which are expected to fail, so this original
part of the implementation, too, is consistent with Beck's definition,
above.  It also shows my own hack fix (SUnit-cmm.116.mcz in the Inbox)
is not a complete and/or proper fix, but it has the correct intention.

We definitely should find a consensus on a fix for this.

Best,
  Chris

[1] - http://swing.fit.cvut.cz/projects/stx/doc/online/english/tools/misc/testfram.htm


More information about the Squeak-dev mailing list