[Vm-dev] Some Performance Numbers: Java vs. CogVM vs. SOM

[Stefan Marr]

Hi Michael:

> On 06 Apr 2015, at 23:06, Michael Haupt <mhaupt at gmail.com> wrote:
> 
>> Am 06.04.2015 um 22:50 schrieb Stefan Marr <smalltalk at stefan-marr.de>:
>> My goal with those benchmarks is to compare how well the just-in-time compilers optimize the code.
> 
> it might seem the pure interpreter comparison adds little value then, but it’s still interesting.

It depends. For the self-optimizing interpreters it is also interesting to see the effect of the optimizations. And, for Sista, and the new bytecode set with quickening, this will also be interesting.
In general, I think interpreter performance is relevant for warmup and average application performance.
But I am still looking for a better way of measuring that more directly.

> Can you share some more insight about the characteristics of the benchmarks with the “surprising" results in interpreter-only mode?

So, the faster than Java benchmarks were DeltaBlue, Json, NBody, Permute, Richards, Storage, Towers.
Honestly, I don’t know whether there is one common characteristic between them. I wouldn’t normally think so.
NBody is mostly object field access and floating point operations. Storage is a tree of arrays, rather a GC benchmark. Permute is very array heavy. It could well be that the Smalltalk bytecode set is just slightly more optimized for the relevant performance critical operations in those benchmarks. However, the average is still about 60% slower. So, on the whole set of benchmarks, the Java interpreter is still better. I would say, overall, it only says that the interpreter is pretty well optimized.


>> The problem here is that not all indexing is pure indexing. Sometimes the indexes are used as values. Getting that right is a pain. So for some benchmarks I gave up after trying to convert to 1-based indexes.
> 
> The indexing base really shouldn’t matter to the compiler.

Right, Graal as well as RPython take care of that nicely. Perhaps Sista will some day, too.

Best regards
Stefan



-- 
Stefan Marr
INRIA Lille - Nord Europe
http://stefan-marr.de/research/