[Vm-dev] Some Performance Numbers: Java vs. CogVM vs. SOM

[Stefan Marr]

Hi Levente:

> On 07 Apr 2015, at 16:58, Levente Uzonyi <leves at elte.hu> wrote:
> 
> On Mon, 6 Apr 2015, Stefan Marr wrote:
> 
>> My goal with those benchmarks is to compare how well the just-in-time compilers optimize the code.
>> It is not a goal to write the most efficient version of the code, taking the known implementation details into account. I want those benchmarks to be reasonably high-level, reasonably idiomatic code.
> 
> In that case maybe the best is to leave everything as it is, but then the results of these benchmarks are hardly comparable between the ports.

Cross-language comparisons are always fishy… However, on platforms with ‘sufficiently smart compilers’, the results are a useful indication for where to look for further optimization potential. On platforms with less extensive optimizations, yes, I agree, one might have the feeling that it is a little unfair.


>>> The PageRank benchmark is so slow that I stopped running it after about 30 minutes. The profiler shows, that it spends over 95% of the time in SomJenkinsRandom class >> #random. I've got a faster (~6x) version of that PRNG, but it's still way to slow. One can consider this as a weakness of the system, but it's also a weak point of the benchmark, that relies so heavily on a fast PRNG implementation. The code is also pretty bad, because it uses only a few bits out of the generated 32, and it has to fight with the signed result. Whoever came up with using that “PRNG" hasn't really put much thought in it...
>> 
>> Yes, the RNG is slow. If you got a faster version, I could surely integrate it thought. As long as it isn’t fundamentally changing things. As mentioned about, for me, the benchmarks are intended to measure how well the optimizers work.
> 
> It's just 6x faster and uses ThirtyTwoBitRegister. If you’re still interested in it, I'll upload it somewhere, along with the missing method of ThirtyTwoBitRegister.

Sure, ThirtyTwoBitRegister is part of Squeak/Pharo, so, seems like a valid approach to me. And, since it is used for performance critical stuff, a compiler writer could decide to recognize it and map it directly on 32bit operations…


>>> About porting:
>>> I don't know what your original goal was, but I don't see why you would keep 0 based indexing in the code. Smalltalk uses 1-based indexing, and this definitely has a negative impact on the Smalltalk results. If you were to port code from Smalltalk to Java, would you keep the 1-based indexes?
>> 
>> The problem here is that not all indexing is pure indexing. Sometimes the indexes are used as values. Getting that right is a pain. So for some benchmarks I gave up after trying to convert to 1-based indexes.
> 
> The methods I found in the PageRank benchmark are easy to port to 1-based index. I did that in my image.

Ok, I think I tried and failed. But perhaps that was another benchmark.

> Well, then you shouldn’t tell Java that the variable is “private static constant", because the VM should be able to find that out. :)

Ehm, well, actually, I even forgot the `final` keyword in the Java version :)
But, there is a difference between representing something as a constant with the available language features, and doing a manual constant propagation. The later is arguably rather bad for code readability and maintainability, because the intension gets lost.

Since we don’t have final fields in Smalltalk, a method returning a literal value is as close as we get. And, it is sufficient for good compilers. Putting a precomputed results in some field, that’s modifiable isn’t something I like. Especially with Smalltalk images and reflection, it’s always a little unclear whether the value is actually what you expect it to be.

Anyway, yes, cross-language and cross-dialect comparisons are inherently problematic.

Thanks for the comments
Stefan

-- 
Stefan Marr
INRIA Lille - Nord Europe
http://stefan-marr.de/research/