[SoC] Project proposal: Improving the testing workflow

[Damien Cassou]

Hi,

I would like to propose the following project for the Google Summer of
Code. Can you please review it and give me feedback?



Unit testing in Squeak


Problematic
-----------

SUnit has some important limitations like:

- When a method is changed it is impossible to know which tests have
to be run. All tests must be launched.
- It is impossible to know if a method validate its tests or not. The
developer must launch all tests and analyze the result.
- The developer never knows if a method or a class is tested or not.
- The framework has no Test entity. Thus, it's impossible to store its
state for example (passing or failing).

Current graphical interfaces have limitations too:

- They never tell you what are the tests for the selected method.
- They force you to create a TestCase class and browse to it when you
want to write a test.
- They do not help you running the tests often.

Often, graphical interfaces use naming conventions to help you write
tests. This is not a good solution and this does not allow for a lot
of features.

Problems to solve
-----------------

- Linkage: it should be possible to know what tests must be run when a
method is modified. This should not be annoying for the developer (no
message box for example). The tests must NOT be run automatically.

- Grouping: it should be possible to group the tests according to
their meaning. For example, a developer should be able to group the
networking tests or the database tests. Moreover, an automatic
categorization can also be done: all failing tests, tests written by a
given developer...

- Launching: it should be possible to run a test or a test group
rapidly and to see the result immediately.

- Duration: an approximate duration of the selected test or group
should be displayed in order to plan the test launching.

- Model and test state: the graphical interface must display the model
and test state (failing or passing) even without a specific request
from the user.

- Storage: the system must be able to save the links between the model
and its tests so that developers of a same team can share information.

Solution
--------

In summer 2006, Karsten Kusche and I developed Dakar Testing to solve
this problems. The work was appreciated and nominated at the ESUG
Innovation Awards 2006
(http://www.esug.org/conferences/14thinternationalsmalltalkconference2006/innovationawards/winnersandnominations/).
However, it has been done in Visual Works, it was a prototype changed
a lot of time and the graphical interface was not portable at all.

Some links about our previous work:

- ESUG innovation award submission:
http://damien.cassou.free.fr/stage_2006/esug06_inno_aw_dakartesting.pdf
- A presentation made for ESUG:
http://damien.cassou.free.fr/stage_2006/presentation/esug_presentation2.pdf
- A French report of my work:
http://damien.cassou.free.fr/stage_2006/rapport_stage_2006/main.pdf

Proposed work
-------------

I propose for this Google Summer of Code to rewrite Dakar Testing from
scratch in Squeak and on top of the OmniBrowser framework.

Related work
------------

Dakar Testing will be the base for this work. However, Stefan
Reichhart work will be used too:
http://smallwiki.unibe.ch/stefanreichhart/

Benefits for the community
--------------------------

I think the current unit-testing workflow is to slow. It takes time to
go from one browser to another, to search for a tests that verify a
given method... Not having basic testing information like test state,
test duration and associations is also a problem. In my opinion, a
better framework and a better GUI will be a benefice for everybody:
developers will write more tests and run them more often and users
will get stabler systems.

Development plan
----------------

End of June: the new framework will be finished and thoroughly tested
End of July: the browser will have basic features, a new pane...
End of August: every ideas will be implemented, a testing report tool
to get global testing information will be finished


-- 
Damien Cassou