Benching #Smalltalk vs Squeak

goran.krampe at bluefish.se goran.krampe at bluefish.se
Tue Nov 25 08:22:42 UTC 2003


"John Brant" <brant at refactory.com> wrote:
> > From: squeak-dev-bounces at lists.squeakfoundation.org [mailto:squeak-dev-
> > bounces at lists.squeakfoundation.org] On Behalf Of goran.krampe at bluefish.se
> > 
> > This weekend I installed .Net 1.1 runtime and Mono-0.28 on my XP laptop.
> > Then I downloaded SharpSmalltalk and tried the benchmarks. Also ran the
> > benchmarks on my 3.7alpha image using the latest Win32 VM 3.6.2. I
> > include the results below.
> > 
> > By an untrained eye it looks like Mono is much slower than MS runtime
> > (but note that I don't know if I should run mono using some neat
> > switches or anything, I just guessed on "mono" :)) - but I still find it
> > nice that it works at all.
> 
> The Mono VM is several times slower than the Microsoft VM running #Smalltalk
> code. I'm not sure, but I'm guessing that some of this time is due to the
> garbage collectors used. I believe that Mono uses the Boehm GC. Since
> #Smalltalk uses boxed SmallIntegers, it really stresses the garbage
> collector. 

Ah. Yes, Mono still uses the Boehm according to the homepage. They write
that they are moving over to Intel's ORP (or something similar, might
have gotten the name wrong).

I also noted that by only typing "mono" thus getting help, there are
switches that supposedly turns optimizations on - but I couldn't get
that to work (it barfed on the switch). Searched around on the net about
that but actually came up empty for once.

> You can see some GC problems in Mono if you try to evaluate large
> factorials. For example, if you evaluate "40000 factorial / 39999
> factorial", it will crash mono, but Microsoft's VM runs it just fine. 

Ok. Even though, neat that it works, right? :) I wonder if it is as slow
on say Linux.
 
> > Also - the MS runtime + SharpSmalltalk has pretty decent numbers
> > compared to Squeak. Half the bytecode speed but faster on sends (though
> > I assume tinyBenchmark is only a hint).
> 
> The bytecode benchmark is essentially showing the #Smalltalk's SmallIntegers
> are really slow. 

Ok.
 
> > If we look at SlopStone (I just used the "Benchmark" package on SM - not
> > sure if it is the same code that SharpSmalltalk includes in its
> > download, but I guess it is) it seems that .Net shines on adding floats
> > (11 times faster), creating objects (7 times faster) and evaluating
> > blocks (31 times faster).
> 
> Yes, both are ports of Bruce Samuelson's benchmarks. I ran the tests a few
> months ago, and #Smalltalk was faster than Squeak on all of the Slopstone
> and Smopstone benchmarks except adding integers and forming sets.

Yes, I only mentioned those where Squeak got really "squashed" since
those might be interesting to understand the reasons why.

> Since
> #Smalltalk uses integer objects instead of tagged integers for
> SmallIntegers, I expect that to be slower. I'm not sure why the forming sets
> test was slower -- it may be due to SmallInteger arithmetic performance.
> 
> 
> > Of course - these numbers look "suspiciously good". :) Smopstone I can't
> > make heads or tails of - and the printouts aren't the same so perhaps
> > the tests have changed too.
> 
> I had to remove a couple tests from the original benchmarks. The sorcerer's
> apprentice was testing rectangle's and #Smalltalk doesn't have any classes
> for rectangles (at least not yet). I also had to remove the generating and
> parsing streams benchmark. Its code was written assuming that a floating
> point number always printed a decimal point, but in #Smalltalk "1.0
> printString = '1'".

Ok - but I also got very confused about the numbers - it didn't seem
like the smopstone number and the execution times are... properly
"related". But I didn't dig into it. :)

For example:

SSt+MS.net
0,6909936     22,8439163546522 generating fractonaccis
0,75108     20,577035735208     generating strings

Squeak
0.499     6.32665330661323     generating fractonaccis
0.187     16.52941176470588     generating strings

So... isn't short time = higher smopstone? For some tests it seems to
be, and not for others.

> > ---------------------------------------------------
> > "Below is running Benchmark using .NET 1.1"
> > C:\SharpSmalltalk\Source\Examples>Benchmarks.exe
> > 49928103 bytecodes/sec; 6900998 sends/sec
> > 
> > Starting benchmarks with repetition count = 16000.
> > 
> > 1000s    time    1000s of
> > itera-   sec-    iterations   slop-
> > tions    onds    per sec      stones   explanation
> > 
> > 3808     0,200288     19013     2,8807002764504     add integers
> > 544     0,0200288     27161     113,170368003408     add floats
> > 960     0,050072     19172     21,737405618902     access strings
> > 320     0,0300432     10651     30,2594566854033     create objects
> > 160     0,0400576     3994     37,3294232006845     copy objects
> > 480     0,200288     2397     5,63893875172692     perform selectors
> 
> Most likely if you ran the benchmarks twice, the perform selectors would be
> much faster the second time. Whenever you perform a selector, the first time
> it is performed, it compiles a block like thing that is directly evaluated
> the next time the selector is performed. I'm guessing that you are seeing
> some bad performance since the classes that .NET uses to generate code
> haven't been jitted yet. I get similar numbers to yours on the other
> benchmarks, but I get 14.1 on the perform selectors.

Ok.

> BTW, these benchmarks should be changed. #Smalltalk's and VisualAge's clocks
> only give 10ms resolution, so most of these benchmarks are close to
> underflowing.
> 
> 
> John Brant

Thanks for the feedback John. And it sure is nice to have another free
Smalltalk on the block!
Especially one that might keep C# at bay. :)

Btw, how do *you* write SharpSmalltalk programs? I mean, what IDE do you
use?

regards, Göran



More information about the Squeak-dev mailing list