[Vm-dev] Some Performance Numbers: Java vs. CogVM vs. SOM

[Levente Uzonyi]

On Mon, 6 Apr 2015, Stefan Marr wrote:

>
> Hi Levente:
>
> My goal with those benchmarks is to compare how well the just-in-time compilers optimize the code.
> It is not a goal to write the most efficient version of the code, taking the known implementation details into account. I want those benchmarks to be reasonably high-level, reasonably idiomatic code.

In that case maybe the best is to leave everything as it is, but then 
the results of these benchmarks are hardly comparable between the ports.

>
>> On 06 Apr 2015, at 22:24, Levente Uzonyi <leves at elte.hu> wrote:
>>
>> I've checked the code, and ran some benchmarks in Squeak. When I tried to load the code, I got complaints from the system, because the name of class variables begin with lowercase letters.
>> ScriptCollector is also missing from Squeak, though it's easy to work this
>> around.
>> There are still plenty of #% sends in the code, which I had to rewrite to #\\.
>
> Yes, I did not commit the Squeak compatible code. It’s only in my image.
> If someone wants it, I could probably upload it somewhere.

AFAIK Eliot is using Squeak for the development of Cog, so I guess he'll 
need it.
Does Pharo have a #% method?

>
>
>> The PageRank benchmark is so slow that I stopped running it after about 30 minutes. The profiler shows, that it spends over 95% of the time in SomJenkinsRandom class >> #random. I've got a faster (~6x) version of that PRNG, but it's still way to slow. One can consider this as a weakness of the system, but it's also a weak point of the benchmark, that relies so heavily on a fast PRNG implementation. The code is also pretty bad, because it uses only a few bits out of the generated 32, and it has to fight with the signed result. Whoever came up with using that “PRNG" hasn't really put much thought in it...
>
> Yes, the RNG is slow. If you got a faster version, I could surely integrate it thought. As long as it isn’t fundamentally changing things. As mentioned about, for me, the benchmarks are intended to measure how well the optimizers work.

It's just 6x faster and uses ThirtyTwoBitRegister. If you're still 
interested in it, I'll upload it somewhere, along with the missing method 
of ThirtyTwoBitRegister.

>
>> About porting:
>> I don't know what your original goal was, but I don't see why you would keep 0 based indexing in the code. Smalltalk uses 1-based indexing, and this definitely has a negative impact on the Smalltalk results. If you were to port code from Smalltalk to Java, would you keep the 1-based indexes?
>
> The problem here is that not all indexing is pure indexing. Sometimes the indexes are used as values. Getting that right is a pain. So for some benchmarks I gave up after trying to convert to 1-based indexes.

The methods I found in the PageRank benchmark are easy to port to 1-based 
index. I did that in my image.

>
>
>> Another thing is about types:
>> [1 - SomPageRank DFactor / n]
>
>> private static double D_FACTOR = 0.85; // damping factor
>>
>> If I were to port this, and I would want to stick to the implementation, then D_FACTOR would be a class variable, and the code would read as: [1.0 - DFactor / n]. But knowing that the constant is not used anywhere else, I see no problem with precalculating the value.
>
> That’s the job of the optimizer/JIT compiler. And, also SOM doesn’t have class variables.
> So, I’ll stick with the current code. That’s a puzzle for Sista to solve. Not something I would want to change in the code.
>
> You might be tempted to do such optimizations, because you know the currently limits of the execution mechanisms. But for me, those benchmarks are about measuring how good the VMs optimize and not, how good you know your VMs.

Well, then you shouldn't tell Java that the variable is "private static 
constant", because the VM should be able to find that out. :)

Levente

>
> Best regards
> Stefan
>
> -- 
> Stefan Marr
> INRIA Lille - Nord Europe
> http://stefan-marr.de/research/
>
>
>
>