[Vm-dev] Some Performance Numbers: Java vs. CogVM vs. SOM

[Eliot Miranda]

Hi Stefan,

On Mon, Apr 6, 2015 at 3:12 AM, Stefan Marr <smalltalk at stefan-marr.de>
wrote:

>
> Hi:
>
> Not sure whether I’ll get to write a little more detailed report, but I
> wanted to briefly share a few pieces of data on the performance of the
> CogVM and StackVM. (Spur benchmarks are still running).
>
> I set up a collection of benchmarks to be able to compare the performance
> of Java, my SOM implementations, and Cog/StackVM [1].
>
> The set contains the following benchmarks:
>  - DeltaBlue
>  - Richards
>  - GraphSearch (search in a graph data structure)
>  - Json (a minimal JSON parser benchmark)
>  - PageRank (a page rank algorithm implementation)
>  -- NBody, Mandelbrot, Bounce, BubbleSort, QuickSort, Fannkuch
>  -- Permute, Queens, Sieve, Storage, Towers
>
> The Java implementations are here [2] and the SOM implementations here [3].
>
> Naturally, the comparison is not ideal between languages. Java isn’t
> Smalltalk, and neither is Pharo/Squeak exactly the same as SOM. However,
> the benchmarks are ported to resemble as closely as possible the
> implementations in the other languages, with an emphasize on
> modern/Smalltalk-ish style where possible. For instance, the DeltaBlue
> implementation in Java is updated to use Java 8 lambdas and other modern
> APIs.
>
> The Results
> ———————————
>
> The most interesting one is peak performance, after warmup, with 100
> iterations of each benchmark. The results are normalized to Java. This
> means, we see the slowdown factors here (less is better). I also report the
> minimal and maximal values to show the range over all benchmarks.
>
>               geomean   min  max
> Java 8          1.0      1.0    1.0
> latest PharoVM 12.9      2.5  182.4 (not sure which exact version of the
> CogVM that is)
> TruffleSOM      2.3      1.0    4.9
> RTruffleSOM     3.0      1.5   11.5
>
> TruffleSOM is SOM implemented as a self-optimizing interpreter on top of
> Truffle, a Java framework.
> RTruffleSOM is SOM as a self-optimizing interpreter on top of RPython’s
> meta-tracing framework (think PyPy).
>
> So, what we see here is that the CogVM is on average 13x slower than Java
> 8. I think that’s not bad at all, considering that it is not doing any
> adaptive compilation yet. The slowest benchmark is PageRank. The fasted one
> is DeltaBlue.
> Compared to the CogVM, my SOM implementations are doing a little better :)
>
>
> Another interesting data point is the pure interpreter performance:
>
>               geomean    min   max
> Java 8 interp   1.0      1.0    1.0
> PharoVM Stack   1.6      0.5   15.3 (not sure which exact version of the
> StackVM that is)
> TruffleSOM      6.3      1.9   15.7
> RTruffleSOM     5.6      1.6   15.7
>
> What we see here is that the StackVM is actually sometimes faster than the
> Java interpreter.
> While the PageRank benchmark is still the slowest, for the following
> benchmarks, the StackVM is faster than Java’s bytecode interpreter:
> DeltaBlue, Json, NBody, Permute, Richards, Storage, Towers.
>
>
> Well, that’s it for the moment.
> I hope that Clement and Eliot find those benchmarks useful, especially for
> the work on Sista.
>

No response to specifics yet because Clément and I are busy with Sista
internals but YES!  Thank you *very much*!  As soon as possible we'll be
looking at this in detail.

And, I wonder whether that makes the SOMs the fasted open source Smalltalk
> implementations? ;)
>
> Best regards
> Stefan
>
>
> [1]
> http://smalltalkhub.com/#!/~StefanMarr/SMark/versions/SOM-Benchmarks-StefanMarr.4
> [2] https://github.com/smarr/Classic-Benchmarks/tree/master/benchmarks/som
> [3] https://github.com/SOM-st/SOM/tree/master/Examples/Benchmarks
>
> --
> Stefan Marr
> INRIA Lille - Nord Europe
> http://stefan-marr.de/research/
>

-- 
best,
Eliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20150406/7abfbec6/attachment.htm