SUnit: Skipping tests?

Markus Gaelli gaelli at emergent.de
Wed Mar 29 09:42:54 UTC 2006


On Mar 28, 2006, at 2:43 AM, Andreas Raab wrote:

> Markus Gaelli wrote:
>>> Like, what if a test which doesn't have any assertion is simply  
>>> not counted? Doesn't make sense to begin with, and then all the  
>>> preconditions need to do is to bail out and the test doesn't  
>>> count...
>> I don't understand this remark within that context.
>> I know a guy who is using that
>>     shouldnt: aBlock raise: anExceptionalEvent : []
>> idiom a lot ;-) , which is good for knowing what is really  
>> tested ;-) but otherwise does not provide any real assertion in  
>> the test. (See most of the BitBltClipBugs tests, which should be  
>> platform independent)
>
> But that is a very valuable assertion! It means that we expect that  
> code to run without an error where (by contradiction) it used to  
> raise an exception.
Let me cite you: ;-)

	self shouldnt:[bb copyBits] raise: Error.

Aehm, isn't any unit test supposed to not throw an Error? So in  
addition to have provided me the information, what was the method  
under test here (which is good :-) )
the only other information you give is that the test was throwing an  
Error. For the hardcore test driven, this should always be the case  
in the beginning, no? ;-)

> And not surprisingly, #shouldnt:raise: (and its other variants) is  
> implemented as an assertion:
>
> shouldnt: aBlock raise: anExceptionalEvent
>   ^self assert: (self executeShould: aBlock inScopeOf:  
> anExceptionalEvent) not
>
Sure.
But: When nil and friends don't understand a thing again, developers  
get yellow failures in statements bracketed like yours and red errors  
for all other statements.
It might well be confusing to get two different results for the same  
error with the only difference that the test developer was saying "no  
errors expected here".

> Makes perfect sense and is a perfectly valid statement for a test.  
> What I was referring to is a test like this:
>
> MyTestCase>>testDummy
>     "Really just a dummy, does nothing"

>
> or like this:
>
> MyTestCase>>windowsTestOnly
>     "A test that only executes on windows"
>     Smalltalk platformName = 'Win32' ifFalse:[^nil].
>     self assert: " ...something... ".
>
> or any other form that does not exercise any assertion in the SUnit  
> framework. It seems to me that such tests are really "empty tests",  
> e.g., by having no assertions there isn't really a statement made  
> whether this test succeeded or not (one might equally claim "it  
> failed" - namely "to test anything"). In any case, I think a good  
> solution would be to simply disregard any tests that don't assert  
> anything.
>
>> Also, tests without any assertions still could execute lots of  
>> methods, which have nice post conditions with them.
>> So besides being good smoke tests, they also could be seen as  
>> tests of that very methods.
>
> If you mean to do this, you could just add a "self assert: true" at  
> the end of it. At least that's a statement that the test is indeed  
> doing something useful, like exercising code and asserting that it  
> really does run through without any other exceptions.
>
  I'd find an empty assert statement at the end of a test confusing,  
too. I also fail to see how this would help a TestRunner  
categorization of skipped tests, unless we tweaked the implementation  
of TestCase >> assert: a bit.

I think what we both want is a visual separation of tests, which have  
assertions, and examples, which don't.
I'd even say that a command, which executes methods having post  
conditions or checking invariants, but does not provide any assertion  
itself, should be treated as a test of all these methods.

As I am also a big fan of examples, I'd really prefer to let  
developers write them without any bad consciousness. Examples are  
smoke tests, the more the merrier, and they should be considered to  
run, without being forced to tag them so. Empty test cases are empty  
and can easily be removed and avoided.

I am all for a test framework which makes that visual difference and  
we are about to build one here roughly based on my taxonomy of unit  
tests:
http://www.iam.unibe.ch/~scg/Archive/Papers/ 
Gael05aTowardsATaxonomyOfUnitTests.pdf

Cheers,

Markus




More information about the Squeak-dev mailing list