On 14-Aug-08, at 5:04 PM, Zulq Alam wrote:

> Is your gut thinking something like this?
>
>  - A test fails
>  - You do something you believe fixes it
>  - The test passes when rerun in different sequence
>  - You move on without actually fixing the bug

It doesn't need to be anything that exotic. If some dependency between  
tests *does* creep in, the shuffling will cause the failure to be  
intermittent. Test B fails only when it comes after Test A, but before  
TestC or something like that. If the suite is shuffled on every run  
it'll be hard to reproduce the failure. A better strategy would be to  
shuffle when the suite is created, then always run in the same order.  
That way you can reproduce the sequence that failed, and figure out  
how to fix it.

Colin