Hi Chris, Andreas,<br><br><div class="gmail_quote">On Thu, May 13, 2010 at 10:57 AM, Chris Muller <span dir="ltr"><<a href="mailto:asqueaker@gmail.com">asqueaker@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Andreas, I have hundreds of tests for Magma and various proprietary<br>
applications that each take anywhere from 1 millisecond to 1 hour.<br>
Would you tell me what I am supposed to do to allow my tests taking<br>
more than 5 seconds not to "timeout"? Please don't say I have to put<br>
a pragma into every single method..<br>
<br>
And please don't say, "tests should be quick." Many of my tests are<br>
written to be _thorough_ in that they ensure adequate volumes of data<br>
are handled correctly. Unlike recently, when a significant<br>
performance regression was introduced 4.1 in the hashed-collections<br>
simply because the test was "quick" and not thorough.<br>
<br>
Tests will take different amounts of time depending on the speed of<br>
the machine running them. So now when I run on a older, slower<br>
machine, I may get test failures just because the hardware wasn't fast<br>
enough, when actually the software is working normally. I see you<br>
have already had to "fiddle" with the timeout value to run through<br>
successfully on whatever machine you are using.<br>
<br>
The reasons stated for introducing this change were:<br>
<br>
> Time out tests by default to avoid a test deadlocking or requiring unexpected user input.<br>
..<br>
> The change heavily simplifies automated testing since tests that deadlock or require user input unexpectedly no longer lock up the testing process but rather fail the individual test.<br>
<br>
After years of straight-away interactive testing, this seems like a<br>
drastic change for a very limited use-case; some kind overnight test<br>
server that spams everyone with test results every morning? Sorry,<br>
I'm just not happy about this..<br>
<br>
What percentage of tests encounter user-input? Machines prompting<br>
users is inherently wrong anyway (like SqueakMap, prompting me to<br>
update the map, it's plain wrong).<br>
<br>
What percentage of tests could possibly encounter a deadlock? That's<br>
for when a test involving multiple processes, right? Small<br>
percentage.<br>
<br>
Therefore, I suggest we make the default timeout much higher, like two<br>
days, and then the very few tests that encounter user-input or<br>
potential deadlocks can stick it in their pragma.<br></blockquote><div><br></div><div>Seems to me the correct default behaviour is that test methods without a timeout are run with an infinite timeout and only methods with a timeout tag will timeout. Any "large" default timeout is kind of arbitrary.</div>
<div><br></div><div>I have to say I love the use of method tags for the test timeout. Its an excellent example of metadata specific to a particular method.</div><div><br></div><div>best</div><div>Eliot</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
- Chris<br>
<br>
<br>
<br>
<br>
On Mon, May 10, 2010 at 10:55 PM, <<a href="mailto:commits@source.squeak.org">commits@source.squeak.org</a>> wrote:<br>
> Andreas Raab uploaded a new version of SUnit to project The Trunk:<br>
> <a href="http://source.squeak.org/trunk/SUnit-ar.76.mcz" target="_blank">http://source.squeak.org/trunk/SUnit-ar.76.mcz</a><br>
><br>
> ==================== Summary ====================<br>
><br>
> Name: SUnit-ar.76<br>
> Author: ar<br>
> Time: 10 May 2010, 8:54:50.15 pm<br>
> UUID: 7b624ddc-b133-894a-a273-7757b0c1b742<br>
> Ancestors: SUnit-ul.75<br>
><br>
> Time out tests by default to avoid a test deadlocking or requiring unexpected user input. The timeout can either be set on a per-class basis (for example DecompilerTests>>defaultTimeout) or using the <timeout: seconds> tag on a per-test basis (for example LocalTest>>testLocaleChanged).<br>
><br>
> The change heavily simplifies automated testing since tests that deadlock or require user input unexpectedly no longer lock up the testing process but rather fail the individual test.<br>
><br>
><br>
> =============== Diff against SUnit-ul.75 ===============<br>
><br>
> Item was changed:<br>
> ----- Method: TestCase>>runCase (in category 'running') -----<br>
> runCase<br>
><br>
> + [[self setUp.<br>
> + self performTest] ensure: [self tearDown]]<br>
> + valueWithin: self timeoutForTest seconds<br>
> + onTimeout:[TestFailure signal: 'Test timed out'].<br>
> + !<br>
> - [self setUp.<br>
> - self performTest] ensure: [self tearDown]<br>
> - !<br>
><br>
> Item was added:<br>
> + ----- Method: SUnitTest>>testTestTimeoutLoop (in category 'testing') -----<br>
> + testTestTimeoutLoop<br>
> + <timeout: 1><br>
> + self should:[[true] whileTrue.] raise: TestFailure.<br>
> + !<br>
><br>
> Item was added:<br>
> + ----- Method: SUnitTest>>testTestTimeoutTag (in category 'testing') -----<br>
> + testTestTimeoutTag<br>
> + <timeout: 1><br>
> + self should:[(Delay forSeconds: 3) wait] raise: TestFailure.<br>
> + !<br>
><br>
> Item was added:<br>
> + ----- Method: TestCase>>timeoutForTest (in category 'accessing') -----<br>
> + timeoutForTest<br>
> + "Answer the timeout to use for this test"<br>
> +<br>
> + | method |<br>
> + method := self class lookupSelector: testSelector asSymbol.<br>
> + (method pragmaAt: #timeout:) ifNotNil:[:tag| ^tag arguments first].<br>
> + ^self defaultTimeout!<br>
><br>
> Item was added:<br>
> + ----- Method: TestCase>>defaultTimeout (in category 'accessing') -----<br>
> + defaultTimeout<br>
> + "Answer the default timeout to use for tests in this test case.<br>
> + The timeout is a value in seconds."<br>
> +<br>
> + ^5 "seconds"!<br>
><br>
> Item was added:<br>
> + ----- Method: SUnitTest>>testTestTimeout (in category 'testing') -----<br>
> + testTestTimeout<br>
> + self should:[(Delay forSeconds: 6) wait] raise: TestFailure.<br>
> + !<br>
><br>
><br>
><br>
<br>
</blockquote></div><br>