New numbers! (was Re: [Vm-dev] Squeak vs Python "smack down")

[Stefan Marr]

Hi Göran:

Those numbers are certainly fun, but wouldn't it be much more interesting to get the SqueakVM and Cog back into the Language Benchmark Game?

http://shootout.alioth.debian.org/

The Dhrystone benchmarks also do not really have the reputation of being, well, interesting, i.e., kind of resemble standard workloads.

Bringing Cog to the Benchmark game would also allow comparison with for instance LuaJIT2.

Best regards
Stefan

On 09 Feb 2011, at 01:50, Göran Krampe wrote:

> Hi!
> 
> So a few small notes were made by a few people on how Sqystone is written, so I tweaked it a bit and these are the more correct numbers, all run with 5000000 loops.
> 
> On 02/08/2011 12:01 AM, Göran Krampe wrote:
>> Cog!
>> ====
>> Squeak 4.2-10966 (soon to be released) + latest Cog r2361 (binary
>> download):
>> Pystone(1.1) time for 50000 passes = 0.06
>> This machine benchmarks at 833333.3 pystones/second
> 
> Pystone(1.1) time for 5000000 passes = 5.448
> This machine benchmarks at 917768.0 pystones/second
> 
>> Regular Squeak
>> ==============
>> Squeak 4.2-10966 (soon to be released) + regular Squeak VM 4.4.7-2357
>> (built from src):
>> Pystone(1.1) time for 50000 passes = 0.503
>> This machine benchmarks at 99403.6 pystones/second
> 
> Pystone(1.1) time for 5000000 passes = 46.448
> This machine benchmarks at 107647.3 pystones/second
> 
>> Regular CPython
>> ===============
>> CPython 3.1.2 (newest in Ubuntu Meerkat, minimal):
>> gokr at quigon:/usr/lib/python3.1/test$ python3.1 pystone.py
>> Pystone(1.1) time for 50000 passes = 0.57
>> This machine benchmarks at 87719.3 pystones/second
>> 
>> NOTE: 3.2 is reportedly a teeny bit faster. Also not built from source.
> 
> gokr at quigon:/usr/lib/python3.1/test$ python3.1 pystone.py 5000000
> Pystone(1.1) time for 5000000 passes = 58.6
> This machine benchmarks at 85324.2 pystones/second
> 
> 
>> 
>> Pypy 1.4
>> ========
>> wget http://pypy.org/download/pypy-1.4.1-linux64.tar.bz2
>> gokr at quigon:~/python/pypy-1.4.1-linux64$ ./bin/pypy
>> ./lib-python/2.5.2/test/pystone.py
>> Pystone(1.1) time for 50000 passes = 0.15
>> This machine benchmarks at 333333 pystones/second
>> gokr at quigon:~/python/pypy-1.4.1-linux64$ ./bin/pypy
>> ./lib-python/2.5.2/test/pystone.py 5000000
>> Pystone(1.1) time for 5000000 passes = 4.8
>> This machine benchmarks at 1.04167e+06 pystones/second
>> 
>> NOTE: Also not built from source. Here we run pystone a second time with
>> 100x more loops and get a substantially better number.
>> 
>> 
>> Shedskin 0.7
>> ============
> > sudo apt-get install g++ libpcre3-dev libgc-dev python-dev
> > sudo dpkg -i shedskin_0.7_all.deb
> > wget http://shedskin.googlecode.com/files/shedskin-examples-0.7.tgz
> > shedskin pystone.py
> > gokr at quigon:~/python/shedskin-examples-0.7$ make
> > g++ -O2 -march=native -fomit-frame-pointer -Wno-deprecated -I.
> > -I/usr/share/shedskin/lib /usr/share/shedskin/lib/builtin.cpp
> > pystone.cpp /usr/share/shedskin/lib/time.cpp
> > /usr/share/shedskin/lib/re.cpp -lgc -lpcre -o pystone
> > gokr at quigon:~/python/gokr at quigon:~/python/shedskin-examples-0.7$ ls -la
> > pystone*
> > -rwxr-xr-x 1 gokr gokr 297329 2011-02-07 23:01 pystone
> > -rw-r--r-- 1 gokr gokr 9193 2011-02-07 23:00 pystone.cpp
> > -rw-r--r-- 1 gokr gokr 1893 2011-02-07 23:00 pystone.hpp
> > -rw-r--r-- 1 gokr gokr 5774 2010-12-11 11:40 pystone.py
> > gokr at quigon:~/python/shedskin-examples-0.7$ ./pystone
> > This machine benchmarks at 2500000.000000 pystones/second
> 
> 5000000 loops gave a more precise number:
> 
> gokr at quigon:~/python/shedskin-examples-0.7$ ./pystone
> This machine benchmarks at 3378378.378378 pystones/second
> 
>> Summary
>> =======
>> 
>> - The regular Squeak VM has not been overrun by CPython in these 6 years
>> time. When I wrote Sqystone Squeak was 5x faster IIRC (can't find the
>> post anymore). Now they are equal more or less, Squeak still a teeny bit
>> faster.
> 
> Actually a bit more (not latest 3.2 Python used though):
> 
> Squeak = 1.28 x CPython
> 
>> - Cog is brutally fast on this one. Compared to CPython and regular
>> Squeak almost 10x faster.
> 
> Cog = 8.4 x Squeak
> Cog = 10.75 x CPython
> 
>> - Pypy is about 20% faster than Cog if given enough time to actually
>> start jitting. Cool for the Pypy project! And cool that they aren't that
>> much faster than Cog. :)
> 
> Cog = 0.88 x Pypy
> 
>> - Shedskin is the "state of the art" of statically compiling Python via
>> C++ using type inferencing etc etc - so I hear. It is said to be faster
>> than Cython and Psyco. It ends up beating Cog, but "only" by a factor of
>> 3x. I say "only" because that seems pretty good to me given that Cog is
>> a JIT and still pretty young and that Shedskin can only run a subset of
>> Python.
> 
> Shedskin = 3.68 x Cog
> 
> regards, Göran

-- 
Stefan Marr
Software Languages Lab
Vrije Universiteit Brussel
Pleinlaan 2 / B-1050 Brussels / Belgium
http://soft.vub.ac.be/~smarr
Phone: +32 2 629 2974
Fax:   +32 2 629 3525