[squeak-dev] SqueakCI Benchmarking

Stefan Marr smalltalk at stefan-marr.de
Tue Feb 26 22:16:41 UTC 2013


Hi Jeff:

On 26 Feb 2013, at 22:54, Jeff Gonis wrote:
> Wow, holy smokes that benchmarking page is impressive.  What sort of
> setup is that running on? Maybe something similar to that for Squeak
> could be my long term goal.

Well, the setup is a bit more elaborate.
I track the RoarVM performance on a number of machines, including a little 64core Tilera manycore chip.

The webapp is Codespeed, a spinoff of the PyPy project [1].

The benchmarks are written with SMark as mentioned earlier.

But they aren't executed by SMark, instead,
ReBench [2] is used to make sure I measure not just noise.
ReBench drives the benchmark execution and starts the VM with the different benchmarks independently.
So, I don't get any funky interactions between different benchmarks which could happen if they were execute during the same VM run.


> As for my rationale for a 5-10 second running time? I wanted something
> that would allow for the image and the VM to become much faster
> without having to adjust the benchmark too much.  Giving us plenty of
> headroom as it were.  That way we could have a single unbroken chain
> of progress on the graph, as things speed up.  I didn't mind
> triggering the GC and such, because I felt that improvements to that
> should be reflected as part of our speed, and I didn't have a specific
> GC benchmark that I could use instead.

Ah, ok, very optimistic ;)
Well, for longterm performance tracking that make sense.

To test for performance regressions, as I do it, I think, more specific/focused benchmarks are another option. Here the longterm aspect isn't as crucial. Instead, the results only need to be comparable to the previous few runs. That way, I have an indication whether I messed something up.

Bye the way, is the slave the benchmarks is run on idle, and only used for those, i.e., it doesn't run any build jobs in parallel?

Best regards
Stefan


[1] https://github.com/tobami/codespeed/
[2] https://github.com/smarr/ReBench

-- 
Stefan Marr
Software Languages Lab
Vrije Universiteit Brussel
Pleinlaan 2 / B-1050 Brussels / Belgium
http://soft.vub.ac.be/~smarr
Phone: +32 2 629 2974
Fax:   +32 2 629 3525



More information about the Squeak-dev mailing list