I have been taking a class in software engineering, One of the only useful lessons to come out of the course is about test coverage.
So I compiled my VM with full test coverage (dude). =P
I fired up this VM and ran the full SM tests with the benchmarks.
When I looked at the output, I found that most source lines had been executed so many times the counters rolled over!
Other lines had been executed hundreds of thousands of times (or just hundreds). Throughout the log there were many examples of lines which had not been executed at all.
Many of these were saftey checks on opcodes.
I know that the Squeak philosophy is to make everything safe. However it seems that they way that the image, compiler, and base clases are laid out effectively makes this redundant.
As CPU pipes get deeper, this could be an issue worth considering...
Are there any coverage test suites for the VM?
What's going on here is that I've started hacking on the VM again and the current version has gone wierd on me... Since I'm compiling with - O0, I might have uncovered another GCC issue too.
With my marginally trustworthy workhorse VM, I get 20 failures and 6 erorrs.
I hack on it and you'd expect more failures and errors.
Well, my latest run gave me 18 failures and 6 errors. However there seemed to be more debug windows and the workspace the benchmarks tried to dump to was in red-box mode. =(
Apparently I still have a mostly working VM except for one or two relatively subtle issues that I want to at least understand before I continue.
On Sep 28, 2004, at 5:57 PM, Alan Grimes wrote:
Are there any coverage test suites for the VM?
There are some test suites for the plugins. It would be nice if "someone" wanted to actually create test suites for *all* the primitives, this would help identify platform specific issues.
Also if we actually did proper parm checking would be good too, I believe we aren't paranoid enough and trust the smalltalk code is passing in the right thing...
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On Sep 28, 2004, at 5:57 PM, Alan Grimes wrote:
Are there any coverage test suites for the VM?
There are some test suites for the plugins. It would be nice if "someone" wanted to actually create test suites for *all* the primitives, this would help identify platform specific issues.
I suppose it would be foolish of me to continue my blind hacking. It would also be of great benefit in general for users to be able to verify that their VM build is functioning properly or to be able to provide the h4x0r$ with the most useful possible information.
To that end, a Squeak VM master specification should be made (does it exist already?)
If no such document already exists, the first step would be to create a general format guideline which specifies how each opcode is to be described.
Next, all opcodes and plug-in calls should be described listing their general purpose, the parameters they accept, a mathematically precice description of their nominal function, and what they are supposed to do if things go horribly, terribly wrong. =P It would also be nice to have a comments section to give additional information about the thinking behind the function and how it might change in the future.
Finally, the coverage suite can probably be a class with a "TestFuncK" method for each call to be tested and a table of constants representing pre-checked values.
I'll get started on this next time I fire up an image....
Alan Grimes writes:
I have been taking a class in software engineering, One of the only useful lessons to come out of the course is about test coverage.
So I compiled my VM with full test coverage (dude). =P
I fired up this VM and ran the full SM tests with the benchmarks.
When I looked at the output, I found that most source lines had been executed so many times the counters rolled over!
I have a test suite that covers most of the interpreter fairly systematically. It is currently targeted at Exupery but if you make the compile methods null ops then it would test the interpreter.
Most of it tests each individual byte code going through all the edge cases.
It's available via SqueakMap and SqueakSource.
Bryce
Hi bryce
do you think that it would make sense to have it for the interpreter? I do :). I contacted Dan because he told me that he fixed the interpreter when doing the 64 port (but dan did not reply yet) and I would like to get those fixes back in the image because I would like to learn by playing with the interpreter and this is really a key asset of Squeak.
Stef
On 29 sept. 04, at 21:41, Bryce Kampjes wrote:
Alan Grimes writes:
I have been taking a class in software engineering, One of the only useful lessons to come out of the course is about test coverage.
So I compiled my VM with full test coverage (dude). =P
I fired up this VM and ran the full SM tests with the benchmarks.
When I looked at the output, I found that most source lines had been executed so many times the counters rolled over!
I have a test suite that covers most of the interpreter fairly systematically. It is currently targeted at Exupery but if you make the compile methods null ops then it would test the interpreter.
Most of it tests each individual byte code going through all the edge cases.
It's available via SqueakMap and SqueakSource.
Bryce
stéphane ducasse writes:
Hi bryce
do you think that it would make sense to have it for the interpreter? I do :). I contacted Dan because he told me that he fixed the interpreter when doing the 64 port (but dan did not reply yet) and I would like to get those fixes back in the image because I would like to learn by playing with the interpreter and this is really a key asset of Squeak.
I think it could make sense for the interpreter. It depends on your working style. I'd want them I if I was working there.
I'll aim to clean the tests up after I get Exupery's tree traversal optimiser working. That could be a few months still, mostly because I'm starting a new job on Monday and moving to a new house the week after.
They would definately help with stabilising interpreter changes. At least to create more of the odd ball crashes early.
Bryce
So can you explain to me what is "but if you make the compile methods null ops then it would test the interpreter." How much would this require?
Stef
On 29 sept. 04, at 22:23, Bryce Kampjes wrote:
stéphane ducasse writes:
Hi bryce
do you think that it would make sense to have it for the interpreter? I do :). I contacted Dan because he told me that he fixed the interpreter when doing the 64 port (but dan did not reply yet) and I would like to get those fixes back in the image because I would like to learn by playing with the interpreter and this is really a key asset of Squeak.
I think it could make sense for the interpreter. It depends on your working style. I'd want them I if I was working there.
I'll aim to clean the tests up after I get Exupery's tree traversal optimiser working. That could be a few months still, mostly because I'm starting a new job on Monday and moving to a new house the week after.
They would definately help with stabilising interpreter changes. At least to create more of the odd ball crashes early.
Bryce
stéphane ducasse writes:
So can you explain to me what is "but if you make the compile methods null ops then it would test the interpreter." How much would this require?
The quick and dirty method would take about five minutes. Just remove the bodies of the methods TestExuperyPlugin>>compile*. They are in the Misc category.
To make the tests work well would involve refactoring them and removing those that rely on a plugin. I've got some extra primitives so I can check that the garbage collector's remembered set has been updated.
There are a few compiler specific tests. They verify that compiled code is being run instead of interpreted code. They would need to be deleted.
To do it properly would involve thinking a little because I'd like to use the same basic suite for both the interpreter and the compiler. It would also involve cleaning them up a little. That would be worthwhile if they were going into the image.
I do plan on cleaning them up. I'll email once we've moved house and I'm ready to start preparing the tests for submission.
Bryce
squeak-dev@lists.squeakfoundation.org