Daniel wrote:

> I think the problem is that tests have diverse domains of validity, 
> and there are neither abstractions nor infrastructure in place to 
> support them.

> Examples:
> - Platform specific tests.
> - Very long running tests, which for most people don't give enough 
> value for their machine time.
> - Non-self-contained tests, for example ones that require external 
> files to be present.
> - Performance tests (only valid on reasonably fast machines. And this 
> might change over time...)
> ...

> Do you know of some set of abstractions/practices/framework to deal 
> with this problem?
Yes! Or at least a step in the right direction  ;-)

SSpec 0.13, which I have recently ported to squeak, is hardwired to 
define a suite of tests using the method category 'specs', whereas as 
you know SUnit effectively hardwires the definition of a suite of tests 
by methods beginning with 'test*'

In order to integrate the two with the same TestRunner I have made steps 
to combine the two with a more generic and flexible solution.

For many years I have been adding code to SUnit that allows a TestCase 
to publish a number of different test suites for different contexts as 
suggested above.

I have now added and extended this feature to SUnit in the hope that it 
may be adopted in 3.9+. I have also extended TestRunner to provide a UI 
for selecting which published suite(s) to run.

All that is needed is to define some conventions for naming suites that 
the community may find useful, including I presume: tests that should 
always pass in an image release, tests that are specific to a particular 
release, tests that illustrate bugs to be addressed, tests that 
highlight when certain fixes have not been loaded or the external 
environment is/is not as expected, long tests, and test suites 
associated with particular products, or specialist releases.

I have also taken the liberty of reorganizing the Class categories from 
SUnit in order to integrate more nicely with SSpec.

I adopted the following Top level categories which I put forward as a 
suggestion for 3.10 if others are agreeable.

Testing-SUnit
Testing-SSpec
Testing-Common
Testing Tests-SUnit
Testing Tests-SSpec

How the scheme works. A TestCase, (or SpecContext) class defines 
#publishedSuites such that

TestCase-c-#publishedSuites
^#( #allStandardTests #longTests #squeak39release #knownBugs 
#allStandardAndLongTests)

each of the nominated published suits defines a match string which 
defines the particular test suite. The match string supports '|' for 
'or' so as to support multiple matches. Also the match string matches 
against both the method name and the method category name together.

<method match string>@<category match string>


TestCase-c-#allStandardTests
^ '*test|*@tests*'

TestCase-c-#longTests
^ '*longtest'

TestCase-c-#allStandardAndLongTests

^ 'test*|longTest*'

The new api for building suites is based upon #suite: . e.g. (myTestCase 
suite: '*@mytests') would return a test suite consisting of all the test 
methods in the category 'mytests'. The testRunner can build a single 
suite across multiple classes by using (TestCase-c-#suite: <match> 
addTo: <suite>) together with the information gathered from 
#publishedSuites.

You will find the code and the test runner in

http://www.squeaksource.com/Testing

I havent finished the spec integration with TestRunner, though SSpec can 
be used with the TextRunner.

enjoy, and do let me know what you think.

Keith




Send instant messages to your online friends http://uk.messenger.yahoo.com