Hi Jeff:

On 26 Feb 2013, at 21:58, Jeff Gonis wrote:

> Hi Everyone,
> 
> So with a lot of help from Frank Shearar and Nicolas Cellier, I have
> introduced performance benchmarking to the SqueakCI server.  You can
> see our current performance trends at the following link:
> http://build.squeak.org/job/SqueakTrunk/performance/

> [..]

> Thank you all for your time, and for any feedback you can provide.
> Jeff

Looks very interesting.

I do something similar for the RoarVM at http://soft.vub.ac.be/~ppp/codespeed/ (page loading times aren't great...)

That's all based on SMark [1] a benchmarking framework in the SUnit-style,
which also properly warms up the JIT on the CogVMs.
You might want to look into it, it has also a number of the Benchmark game benchmarks.

Another thing, what's your rational for choosing  5-10sec runtimes?

I try typically to make the runtime just long enough to avoid an impact of imprecise time measurement.
Most of the time, the goal is to keep the runtime low and avoid triggering GC, except when GC is supposed to be measured.

Best regards
Stefan


[1] http://smalltalkhub.com/#!/~StefanMarr/SMark

-- 
Stefan Marr
Software Languages Lab
Vrije Universiteit Brussel
Pleinlaan 2 / B-1050 Brussels / Belgium
http://soft.vub.ac.be/~smarr
Phone: +32 2 629 2974
Fax:   +32 2 629 3525