On 11/1/06, Daniel Vainsencher <daniel.vainsencher at gmail.com> wrote:
> Do you know of some set of abstractions/practices/framework to deal with
> this problem?

Sure.  Mostly practices.  I think we have enough abstractions and
frameworks already, though they could be better.

Divide tests into the ones that you expect to be run by people
developing other packages, and those run by developers of your
package.  The first are going to be included with your package, and
the second will be in a separate package in MC.

There is a base image that includes some set of packages.  Other
packages are said to "work with the base image".  The tests in the
base image all run on all platforms.  There might be platform specific
tests, but they are not in the base image.  If a package works with
the base image then when you load it, all of the original tests in the
image will work, and the tests with the package will work.
Presumably, the private tests for the package will work, too, but that
is up to the developer of the package.

Of course, just because two packages work with the base image does not
mean that they will work with each other.  It makes sense to have a
"universe" in which any combination of packages in the universe will
work with each other, as well as with the base image.  This takes more
testing and certification, and there has to be a "universe maintainer"
who does this.  In theory it is easy, in practice it is a lot of work.
 But much of the work can be automated.  We can worry about this after
we have a base image in which all tests work.  The first priority is
to create a world in which developers can assume that any broken tests
are their fault.

The current set of tests in Squeak are not too bad.  I think they will
take on the order of half an hour on a fast machine, so it is possible
for a developer to run them all before releasing code.  People do not
run all the tests every time they make a little change, no matter what
the books say.  People tend to pick the most relevant test suites and
run them after each little change, so those tests will run in  just a
few minutes.  They don't run long tests very often.  So, I do not
think that speed is the problem.

The problem is that all the tests is a relase-image should work.
Period.  Either fix them or remove them.  From then on, if someone
offers a patch and the patch breaks some tests, reject the patch.
Never release an image with broken tests.  Don't accept code that
breaks tests unless you are trying to help the author and plan to fix
the tests yourself.

-Ralph