Making sense of VM test coverage.

Alan Grimes alangrimes at starpower.net
Wed Sep 29 00:57:27 UTC 2004


I have been taking a class in software engineering, One of the only 
useful lessons to come out of the course is about test coverage.

So I compiled my VM with full test coverage (dude). =P 

I fired up this VM and ran the full SM tests with the benchmarks. 

When I looked at the output, I found that most source lines had been 
executed so many times the counters rolled over!

Other lines had been executed hundreds of thousands of times (or just 
hundreds). Throughout the log there were many examples of lines which 
had not been executed at all. 

Many of these were saftey checks on opcodes. 

I know that the Squeak philosophy is to make everything safe. However 
it seems that they way that the image, compiler, and base clases are 
laid out effectively makes this redundant. 

As CPU pipes get deeper, this could be an issue worth considering... 

Are there any coverage test suites for the VM? 

What's going on here is that I've started hacking on the VM again and 
the current version has gone wierd on me... Since I'm compiling with -
O0, I might have uncovered another GCC issue too.

With my marginally trustworthy workhorse VM, I get 20 failures and 6 
erorrs. 

I hack on it and you'd expect more failures and errors. 

Well, my latest run gave me 18 failures and 6 errors. However there 
seemed to be more debug windows and the workspace the benchmarks tried 
to dump to was in red-box mode. =( 

Apparently I still have a mostly working VM except for one or two 
relatively subtle issues that I want to at least understand before I 
continue. 



More information about the Squeak-dev mailing list