[ANN] Nile 0.9.0
Andreas Raab
andreas.raab at gmx.de
Sat Jun 9 20:09:18 UTC 2007
Hi Damien -
Interesting benchmarks. I think there is something to be learned from
running micro benchmarks so here are a few comments:
Damien Cassou wrote:
> next
> Squeak: 41.5 operations/sec
> Nile: 45
> 8% faster
On my machine:
Squeak result: 39.1
Nile result: 32.2
Comparison: -18%
The real "lesson" of this benchmark is to only trust your micro
benchmark as far as you can throw them ;-) Given that both
implementation use primitiveNext and given that the code is designed to
guarantee a hit in the VM's at-cache there shouldn't be any difference
whatsoever.
The difference we're seeing here (which made me investigate the matter
more closely) should also be an indicator that there may be something
wrong with the benchmarking process. Running it repeatedly gives:
Squeak: 38.9 35.6 30.5 39.2 31.8
Nile: 36.2 33.9 37.9 40.8 31.3
Delta: -7% -5% 19% 4% -1%
meaning there is a 20% difference within five runs which seems extremely
high for a microbenchmark that's just a primitive call. I think you'll
have to fix the benchmarks to give more consistent results if you want
to make any claims about relative speed improvements.
> next:
> Squeak: 98.2
> Nile: 158.2
> 38% faster
On my machine:
Squeak result: 90.3
Nile result: 130.5
Comparison: 31%
The lesson here is (at least for me) that after trying to wrap my brain
around the code in NSStringReader>>next: I'm willing to give up the
speed. A nice example for what can be done if you understand byte code
execution but by no means production code (pity the bugger who at some
point will need to understand that the "0-position" is required since
position cannot occur on the right-hand side of that expression ;-)
Oh, and of course "no comments == not helpful" in particular when it
comes to that level of optimization. And while having tests is great,
having 73 out of 87 classes without a single line of explanation (class
comment) is pretty pathetic.
> nextPut:
> Squeak: 24.9
> Nile: 42.4
> 41% faster
On my machine:
Squeak result: 44.8
Nile result: 71.4
Comparison: 37%
Oddly, this benchmark scores the same with or without the primitive in
WriteStream>>nextPut: ... which is pretty strange if you ask me. I have
a suspicion that the primitiveNextPut hasn't been used in a long time
and may need to be rewhacked to perform properly.
> nextPutAll:
> Squeak: 115.9
> Nile: 120.2
> 4% faster
On my machine:
Squeak result: 117.8
Nile result: 114.8
Comparison: -3%
Not really much of a lesson here other than if both implementations take
"reasonable care" they'll likely end up with similar speed.
So all in all, interesting benchmarks but you need to fix the variation
issue. With the variations we're seeing, all but the nextPut: benchmark
(which I suspect suffers from a broken nextPut: primitive) could fall
either way so there isn't really much of claim to be made here.
Cheers,
- Andreas
More information about the Squeak-dev
mailing list
|