[Vm-dev] some stupid failures

Marcel Taeumel marcel.taeumel at hpi.de
Tue Jan 5 08:22:02 UTC 2021


Hi Nicolas.

> Do we really want to keep this kind of test?

Such benchmarks (and benchmark-like tests) should at least average over several runs and only fail as a test if something actually got slower on average. Or something like that. A single misbehaving run should not be the reason for such a test failure.

Maybe we can tweak #should:notTakeMoreThan: to evaluate the block several times? But then it cannot fail early on as it is doing now ... Hmmm...

Best,
Marcel
Am 05.01.2021 09:08:46 schrieb Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com>:

Hi all,
sometimes, some build fail for just 1 test...

Here https://travis-ci.com/github/OpenSmalltalk/opensmalltalk-vm/jobs/468407844
a squeak.stack.v3

RenderBugz
✗ #testSetForward (7ms)
TestFailure: Block evaluation took more than the expected 0:00:00:00.004
RenderBugz(TestCase)>>assert:description:
RenderBugz(TestCase)>>should:notTakeMoreThan:
RenderBugz(TestCase)>>should:notTakeMoreThanMilliseconds:
RenderBugz>>shouldntTakeLong:
RenderBugz>>testSetForward ...shouldntTakeLong: [ t forwardDirection: 180.0 .
self assert: ( t forwardDirection = 0.0 ) ]
RenderBugz(TestCase)>>performTest

4ms, really? On C.I. infrastructure, anything can happen...
Do we really want to keep this kind of test?
We eventually could once startup performance is known (see
isLowerPerformance discussion on squeak-dev), but in the interim, I
suggest we neutralize this specific test in Smalltalk-CI.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20210105/9a1a25b8/attachment.html>


More information about the Vm-dev mailing list