On 26 February 2013 22:16, Stefan Marr <smalltalk at stefan-marr.de> wrote:
> Hi Jeff:
>
> On 26 Feb 2013, at 22:54, Jeff Gonis wrote:
>> Wow, holy smokes that benchmarking page is impressive.  What sort of
>> setup is that running on? Maybe something similar to that for Squeak
>> could be my long term goal.
>
> Well, the setup is a bit more elaborate.
> I track the RoarVM performance on a number of machines, including a little 64core Tilera manycore chip.
>
> The webapp is Codespeed, a spinoff of the PyPy project [1].
>
> The benchmarks are written with SMark as mentioned earlier.
>
> But they aren't executed by SMark, instead,
> ReBench [2] is used to make sure I measure not just noise.
> ReBench drives the benchmark execution and starts the VM with the different benchmarks independently.
> So, I don't get any funky interactions between different benchmarks which could happen if they were execute during the same VM run.
>
>
>> As for my rationale for a 5-10 second running time? I wanted something
>> that would allow for the image and the VM to become much faster
>> without having to adjust the benchmark too much.  Giving us plenty of
>> headroom as it were.  That way we could have a single unbroken chain
>> of progress on the graph, as things speed up.  I didn't mind
>> triggering the GC and such, because I felt that improvements to that
>> should be reflected as part of our speed, and I didn't have a specific
>> GC benchmark that I could use instead.
>
> Ah, ok, very optimistic ;)
> Well, for longterm performance tracking that make sense.
>
> To test for performance regressions, as I do it, I think, more specific/focused benchmarks are another option. Here the longterm aspect isn't as crucial. Instead, the results only need to be comparable to the previous few runs. That way, I have an indication whether I messed something up.
>
> Bye the way, is the slave the benchmarks is run on idle, and only used for those, i.e., it doesn't run any build jobs in parallel?

The slave can run two concurrent builds (so yes, the benchmark
accuracy is compromised). Otherwise, as far as I know at least, the
box only serves up Jenkins. Chris Cunnington or Ken Causey could give
more detailed answers: I just keep adding CI jobs to the thing.

frank

> Best regards
> Stefan
>
>
> [1] https://github.com/tobami/codespeed/
> [2] https://github.com/smarr/ReBench
>
> --
> Stefan Marr
> Software Languages Lab
> Vrije Universiteit Brussel
> Pleinlaan 2 / B-1050 Brussels / Belgium
> http://soft.vub.ac.be/~smarr
> Phone: +32 2 629 2974
> Fax:   +32 2 629 3525
>
>