Tests and software process

Ralph Johnson johnson at cs.uiuc.edu
Wed Nov 1 13:30:30 UTC 2006

Squeak comes with a large set of SUnit tests.  Unfortunately, some of
them don't work.  As far as I can tell, there is NO recent version of
Squak in which all the tests work.

This is a sign that something is wrong.  The main purpose of shipping
tests with code is so that people making changes can tell when they
break things.  If the tests don't work then people will not run them.
If they don't run the tests then the tests are useless.  The current
set of tests are useless because of the bad tests.  Nobody complains
about them, which tells me that nobody runs them.  So, it is all a
waste of time.

If the tests worked then it would be easy to make a new version.
Every bug fix would have to come with a test that illustrates the bug
and shows that it has been fixed.  The group that makes a new version
would check that all tests continue to work after the bug fix.

An easy way to make all the tests run is to delete the ones that don't
work.  There are thousands of working tests and, depending on the
version, dozens of non-working tests.  Perhaps the non-working tests
indicate bugs, perhaps they indicate bad tests.  It seems a shame to
delete tests that are illustrating bugs.  But if these tests don't
work, they keep the other tests from being useful.  Programmers need
to know that all the tests worked in the virgin image, and that if the
tests quit working, it is there own fault.

No development image should ever be shipped with any failing tests.

-Ralph Johnson

More information about the Squeak-dev mailing list