[Vm-dev] some stupid failures

Marcel Taeumel marcel.taeumel at hpi.de
Tue Jan 5 08:22:02 UTC 2021

Hi Nicolas.

> Do we really want to keep this kind of test?

Such benchmarks (and benchmark-like tests) should at least average over several runs and only fail as a test if something actually got slower on average. Or something like that. A single misbehaving run should not be the reason for such a test failure.

Maybe we can tweak #should:notTakeMoreThan: to evaluate the block several times? But then it cannot fail early on as it is doing now ... Hmmm...

Am 05.01.2021 09:08:46 schrieb Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>:

Hi all,
sometimes, some build fail for just 1 test...

Here https://travis-ci.com/github/OpenSmalltalk/opensmalltalk-vm/jobs/468407844
a squeak.stack.v3

✗ #testSetForward (7ms)
TestFailure: Block evaluation took more than the expected 0:00:00:00.004
RenderBugz>>testSetForward ...shouldntTakeLong: [ t forwardDirection: 180.0 .
self assert: ( t forwardDirection = 0.0 ) ]

4ms, really? On C.I. infrastructure, anything can happen...
Do we really want to keep this kind of test?
We eventually could once startup performance is known (see
isLowerPerformance discussion on squeak-dev), but in the interim, I
suggest we neutralize this specific test in Smalltalk-CI.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20210105/9a1a25b8/attachment.html>

More information about the Vm-dev mailing list