Folks, SUnits are a style of testing that verify the operation of a list of specific methods. The structure for SUnits has been in Squeak for a while, but only forty (40) out of 44,000 methods actually have a validating test. I have embarked on a radical program to improve that. Here is how I have changed things:
[ ] To test a method, you needs to make up a receiver, arguments, and a known answer. Scott Wallace is introducing an "Instance Browser" to Squeak that shows you an example receiver and arguments for each method. It lets you run the example. But, where are we going to get the example receivers and arguments for a method shown in an Instance Browser? They can be the same as for a test of the method. I have written code that generates objects that can be used for *either* an SUnit test, or an example method call.
[ ] Another thing that should be associated with each method is some "type information". This is the protocol that each argument object needs to respond to. The same mechanism that produces a test case can store this info also.
[ ] In classic SUnits, there is a separate 'testing class' for each class in Squeak. This would double the number of classes, and is too much overhead. I have created a new subclass of TestCase that owns a Verifier. Verifier is a subclass MessageSend that holds the answer that the method is supposed to produce.
Object #() MessageSend #('receiver' 'selector' 'arguments') MethodCall #('lastValue' 'methodInterface' 'timeStamp') Verifier #('desiredAnswer' 'passed') VerifierOfProperty #('theTest')
I have made a subclass of TestCase that holds a Verifier. A TestSuite of these behaves like a normal test suite, and does not do too much violence to the careful design of SUnits.
[ ] A Verifier can only test (a) whether a method returns the correct value and (b) whether it gets an error while running. A VerifierOfProperty tests for a specific state change after a method has run. Classic SUnits allow other kinds of assertions and tests. Except for verifying that a method produces a certain kind of error in a certain case, a series of VerifierOfProperty can test everything that a classic TestCase can.
[ ] To collect the test cases, I simply enumerate the implementors of #exampleFor:, and call each one with 'all'. That returns a list of Verifiers, which are run as test cases.
[ ] The SUnit code comes with a TestRunner user interface. Say TestRunner open and click on Run All. After you have my update, the TestCase called TestViaMethodCall collects and runs all tests that are created with #exampleFor:. (One drawback is that when a method has more than one test, the individual subtests are not named. We could give them names in the future.)
These changes will be in the next batch of external updates. If you can't wait, ask me for the file now. I invite you to supply an example for any method by implementing #exampleFor: in some class.
(In the future, we'd like to have the kind of TestCases that SUnit has now, but not have the overhead of doubling the number of classes. One way to do this is to make Metaclass do everything that TestCase does. The individual testing methods would begin with a standard prefix like 'textSU'. Any class that has a method whose name begins with 'textSU' would be found by TestRunner, and entered into its list. Any comments on this?)
--Ted.
This is great. I'm eager to play with it. I was afraid that the Squeak community would miss one of the really important values spread by XP: testing. So tool support was missing but with your changes we will be able to push test a step further.
Thanks for that!
on 11/30/01 8:32 PM, Ted Kaehler at Ted@SqueakLand.org wrote:
Folks, SUnits are a style of testing that verify the operation of a list of specific methods. The structure for SUnits has been in Squeak for a while, but only forty (40) out of 44,000 methods actually have a validating test. I have embarked on a radical program to improve that. Here is how I have changed things:
[ ] To test a method, you needs to make up a receiver, arguments, and a known answer. Scott Wallace is introducing an "Instance Browser" to Squeak that shows you an example receiver and arguments for each method. It lets you run the example. But, where are we going to get the example receivers and arguments for a method shown in an Instance Browser? They can be the same as for a test of the method. I have written code that generates objects that can be used for *either* an SUnit test, or an example method call.
[ ] Another thing that should be associated with each method is some "type information". This is the protocol that each argument object needs to respond to. The same mechanism that produces a test case can store this info also.
[ ] In classic SUnits, there is a separate 'testing class' for each class in Squeak. This would double the number of classes, and is too much overhead. I have created a new subclass of TestCase that owns a Verifier. Verifier is a subclass MessageSend that holds the answer that the method is supposed to produce.
Object #() MessageSend #('receiver' 'selector' 'arguments') MethodCall #('lastValue' 'methodInterface' 'timeStamp') Verifier #('desiredAnswer' 'passed') VerifierOfProperty #('theTest')
I have made a subclass of TestCase that holds a Verifier. A TestSuite of these behaves like a normal test suite, and does not do too much violence to the careful design of SUnits.
[ ] A Verifier can only test (a) whether a method returns the correct value and (b) whether it gets an error while running. A VerifierOfProperty tests for a specific state change after a method has run. Classic SUnits allow other kinds of assertions and tests. Except for verifying that a method produces a certain kind of error in a certain case, a series of VerifierOfProperty can test everything that a classic TestCase can.
[ ] To collect the test cases, I simply enumerate the implementors of #exampleFor:, and call each one with 'all'. That returns a list of Verifiers, which are run as test cases.
[ ] The SUnit code comes with a TestRunner user interface. Say TestRunner open and click on Run All. After you have my update, the TestCase called TestViaMethodCall collects and runs all tests that are created with #exampleFor:. (One drawback is that when a method has more than one test, the individual subtests are not named. We could give them names in the future.)
These changes will be in the next batch of external updates. If you can't wait, ask me for the file now. I invite you to supply an example for any method by implementing #exampleFor: in some class.
(In the future, we'd like to have the kind of TestCases that SUnit has now, but not have the overhead of doubling the number of classes. One way to do this is to make Metaclass do everything that TestCase does. The individual testing methods would begin with a standard prefix like 'textSU'. Any class that has a method whose name begins with 'textSU' would be found by TestRunner, and entered into its list. Any comments on this?)
--Ted.
At 11:32 -0800 2001.11.30, Ted Kaehler wrote:
Ted,
I really do applaud your efforts to include testing into the Squeak culture. The lack of comprehensive testing has really slowed us down, since every new feature has broken something old, but we never found about about this in a timely fashion.
I didn't follow the details of all of the changes that you propose. But I endorse the principle. An example would make it much easier to understand what you have done, I think.
(In the future, we'd like to have the kind of TestCases that SUnit has now, but not have the overhead of doubling the number of classes. One way to do this is to make Metaclass do everything that TestCase does. The individual testing methods would begin with a standard prefix like 'textSU'. Any class that has a method whose name begins with 'textSU' would be found by TestRunner, and entered into its list. Any comments on this?)
How about having a method on a class that answers a collection of method selectors that are the tests for instances of that class? The inherited implementation could select all of the methods that contain a particular prefix, but any class could override this method to do something different, if desired.
Andrew
[ ] In classic SUnits, there is a separate 'testing class' for each class in Squeak. This would double the number of classes, and is too much overhead. I have created a new subclass of TestCase that owns a Verifier. Verifier is a subclass MessageSend that holds the answer that the method is supposed to produce.
Was it intended that a testing class exist for every other class in the system? Most uses of Sunit that I've seen will have a TestCase class for a subsystem, not an individual class. For example, with Kats, I have a suite of 45 tests in one class that exercises the classes and protocols designed for use by the users of Kats. I don't usually directly test the classes and protocols that are internal to a package, those will get tested indirectly via the tests of the public interface of the package. Of course, it doesn't hurt to have tests for everything.
It sounds like this enhancement will make it much easier to add tests to a system. I can't wait to try it out.
- Stephen
Folks, Yes, I do agree with this:
At 10:17 PM -0500 11/30/01, Stephen Pair wrote:
Most uses of Sunit that I've seen will have a TestCase class for a subsystem, not an individual class. For example, with Kats, I have a suite of 45 tests in one class that exercises the classes and protocols designed for use by the users of Kats. I don't usually directly test the classes and protocols that are internal to a package, those will get tested indirectly via the tests of the public interface of the package.
At 8:48 AM -0800 12/3/01, Ward Cunningham wrote:
[snip] That is not to say that some language/environment innovation couldn't change this statistic. I've had good luck with tables of values interpreted by fixture code that is much smaller than the typical SUnit equivalent.
Scott Wallace has suggested a similar table driven way of generating the examples. I think #exampleFor: is tolerable, and anyone could implement it to read a table.
Perhaps the match-up of test cases and examples of methods is not perfect. TestCases tend to be testing large subsystems, and examples for users will start out with the simple 'beginner' methods. But some methods will be both need legitimate testing and be good for users to look at, such as String>>asNumber and Base64MimeConverter>>mimeEncode:.
4568ViaSUnit-tk.cs 4580ViaSUnit-2-tk.cs
I have added a second update that makes TestRunner show one entry per class that has the MethodCall type of test. Any class-object that responds to #exampleFor: will be listed here. The intention is to keep traditional TestCases, and also the newer TestViaMethodCalls. With this modification, the two ways of specifying test cases can live side by side.
I really appreciate everyone who wrote to me to educate me about SUnits.
--Ted.
Ted, thanks for tackling this. I look forward to trying out your code.
I like the idea of connecting examples and tests and I thought I'd mention python's doctest framework. There, examples are embedded in the documentation strings within the code, along with the output that would appear in an interactive session; these can also be run as tests. Eg
""" addOne is used as follows:
addOne(1)
2
addOne('')
None
(run these tests with "doctest file.py") """
I find this makes a nice complement to the heavier sunit style of test framework. Maybe it would be useful in squeak, maybe not. I tried to imagine how it would fit..
For one thing there's no existing recognizable '>>>' prompt to flag code snippets. For another, the printed response to many squeak messages will be "an ObjectOfSomeKind (someunpredictablenumber)". To make tests like these work, I guess one could assume a python-like read-eval-print loop with the instance numbers stripped out.
Regards, -Simon
Ted Kaehler wrote:
(In the future, we'd like to have the kind of TestCases that SUnit has now, but not have the overhead of doubling the number of classes. One way to do this is to make Metaclass do everything that TestCase does.
Three (possibly contradictory) thoughts come to mind ...
We find that xp's test first method tends to produce about as much test code as code under test. Is it then unreasonable to expect the number of classes to double in proportion? Also, the author of a test sometimes finds need for instance variables and helper methods. Let's not loose these in an attempt to optimize.
That is not to say that some language/environment innovation couldn't change this statistic. I've had good luck with tables of values interpreted by fixture code that is much smaller than the typical SUnit equivalent.
Finally, if Tests are a parallel hierarchy, then perhaps we should take a cue from the Class-MetaClass parallel hierarchy. This would suggest that the buttons on the browser should read: [instance][class][test]
-- Ward Cunningham v 503-245-5633 mailto:ward@c2.com f 503-246-5587 http://c2.com/
squeak-dev@lists.squeakfoundation.org
-
Andrew P. Black -
ducasse -
Simon Michael -
Stephen Pair -
Ted Kaehler -
Ward Cunningham