[ANN] Nile 0.9.0

Andreas Raab andreas.raab at gmx.de
Sat Jun 9 20:09:18 UTC 2007

Hi Damien -

Interesting benchmarks. I think there is something to be learned from 
running micro benchmarks so here are a few comments:

Damien Cassou wrote:
> next
>    Squeak: 41.5 operations/sec
>    Nile:       45
>    8% faster

On my machine:

	Squeak result: 39.1
	Nile result: 32.2
	Comparison: -18%

The real "lesson" of this benchmark is to only trust your micro 
benchmark as far as you can throw them ;-) Given that both 
implementation use primitiveNext and given that the code is designed to 
guarantee a hit in the VM's at-cache there shouldn't be any difference 

The difference we're seeing here (which made me investigate the matter 
more closely) should also be an indicator that there may be something 
wrong with the benchmarking process. Running it repeatedly gives:

Squeak: 38.9  35.6  30.5  39.2  31.8
Nile:   36.2  33.9  37.9  40.8  31.3
Delta:   -7%   -5%   19%    4%   -1%

meaning there is a 20% difference within five runs which seems extremely 
high for a microbenchmark that's just a primitive call. I think you'll 
have to fix the benchmarks to give more consistent results if you want 
to make any claims about relative speed improvements.

> next:
>    Squeak: 98.2
>    Nile:     158.2
>    38% faster

On my machine:

	Squeak result: 90.3
	Nile result: 130.5
	Comparison: 31%

The lesson here is (at least for me) that after trying to wrap my brain 
around the code in NSStringReader>>next: I'm willing to give up the 
speed. A nice example for what can be done if you understand byte code 
execution but by no means production code (pity the bugger who at some 
point will need to understand that the "0-position" is required since 
position cannot occur on the right-hand side of that expression ;-)

Oh, and of course "no comments == not helpful" in particular when it 
comes to that level of optimization. And while having tests is great, 
having 73 out of 87 classes without a single line of explanation (class 
comment) is pretty pathetic.

> nextPut:
>    Squeak: 24.9
>    Nile:       42.4
>    41% faster

On my machine:

	Squeak result: 44.8
	Nile result: 71.4
	Comparison: 37%

Oddly, this benchmark scores the same with or without the primitive in 
WriteStream>>nextPut: ... which is pretty strange if you ask me. I have 
a suspicion that the primitiveNextPut hasn't been used in a long time 
and may need to be rewhacked to perform properly.

> nextPutAll:
>    Squeak: 115.9
>    Nile: 120.2
>    4% faster

On my machine:

	Squeak result: 117.8
	Nile result: 114.8
	Comparison: -3%

Not really much of a lesson here other than if both implementations take 
"reasonable care" they'll likely end up with similar speed.

So all in all, interesting benchmarks but you need to fix the variation 
issue. With the variations we're seeing, all but the nextPut: benchmark 
(which I suspect suffers from a broken nextPut: primitive) could fall 
either way so there isn't really much of claim to be made here.

   - Andreas

More information about the Squeak-dev mailing list