[Vm-dev] Some Performance Numbers: Java vs. CogVM vs. SOM

Stefan Marr smalltalk at stefan-marr.de
Mon Apr 6 10:12:54 UTC 2015


Hi:

Not sure whether I’ll get to write a little more detailed report, but I wanted to briefly share a few pieces of data on the performance of the CogVM and StackVM. (Spur benchmarks are still running).

I set up a collection of benchmarks to be able to compare the performance of Java, my SOM implementations, and Cog/StackVM [1].

The set contains the following benchmarks:
 - DeltaBlue
 - Richards
 - GraphSearch (search in a graph data structure)
 - Json (a minimal JSON parser benchmark)
 - PageRank (a page rank algorithm implementation)
 -- NBody, Mandelbrot, Bounce, BubbleSort, QuickSort, Fannkuch
 -- Permute, Queens, Sieve, Storage, Towers

The Java implementations are here [2] and the SOM implementations here [3].

Naturally, the comparison is not ideal between languages. Java isn’t Smalltalk, and neither is Pharo/Squeak exactly the same as SOM. However, the benchmarks are ported to resemble as closely as possible the implementations in the other languages, with an emphasize on modern/Smalltalk-ish style where possible. For instance, the DeltaBlue implementation in Java is updated to use Java 8 lambdas and other modern APIs.

The Results
———————————

The most interesting one is peak performance, after warmup, with 100 iterations of each benchmark. The results are normalized to Java. This means, we see the slowdown factors here (less is better). I also report the minimal and maximal values to show the range over all benchmarks.

              geomean   min  max
Java 8          1.0      1.0    1.0
latest PharoVM 12.9      2.5  182.4 (not sure which exact version of the CogVM that is)
TruffleSOM      2.3      1.0    4.9
RTruffleSOM     3.0      1.5   11.5

TruffleSOM is SOM implemented as a self-optimizing interpreter on top of Truffle, a Java framework.
RTruffleSOM is SOM as a self-optimizing interpreter on top of RPython’s meta-tracing framework (think PyPy).

So, what we see here is that the CogVM is on average 13x slower than Java 8. I think that’s not bad at all, considering that it is not doing any adaptive compilation yet. The slowest benchmark is PageRank. The fasted one is DeltaBlue.
Compared to the CogVM, my SOM implementations are doing a little better :)


Another interesting data point is the pure interpreter performance:

              geomean    min   max
Java 8 interp   1.0      1.0    1.0
PharoVM Stack   1.6      0.5   15.3 (not sure which exact version of the StackVM that is)
TruffleSOM      6.3      1.9   15.7
RTruffleSOM     5.6      1.6   15.7

What we see here is that the StackVM is actually sometimes faster than the Java interpreter.
While the PageRank benchmark is still the slowest, for the following benchmarks, the StackVM is faster than Java’s bytecode interpreter: DeltaBlue, Json, NBody, Permute, Richards, Storage, Towers.


Well, that’s it for the moment.
I hope that Clement and Eliot find those benchmarks useful, especially for the work on Sista. 


And, I wonder whether that makes the SOMs the fasted open source Smalltalk implementations? ;)

Best regards
Stefan


[1] http://smalltalkhub.com/#!/~StefanMarr/SMark/versions/SOM-Benchmarks-StefanMarr.4
[2] https://github.com/smarr/Classic-Benchmarks/tree/master/benchmarks/som
[3] https://github.com/SOM-st/SOM/tree/master/Examples/Benchmarks

-- 
Stefan Marr
INRIA Lille - Nord Europe
http://stefan-marr.de/research/





More information about the Vm-dev mailing list