[squeak-dev] A Benchmarking tool for the trunk?

David T. Lewis lewis at mail.msen.com
Wed Apr 27 23:59:46 UTC 2016


On Wed, Apr 27, 2016 at 09:37:29PM +0200, Tim Felgentreff wrote:
> Hi,
> 
> We have a proposal for a tool that we think might be useful to have in
> trunk.

This looks very nice! May I suggest that you (or we) create a SqueakMap
entry for this, so that it can be easily located and loaded in Squeak
5.0 and trunk images? If it also works in Pharo, someone will probably
volunteer to make a ConfigurationOfBenchmark also.

There are lots of advantages to maintaining a package like this as an
external package, just as long as the package is easy to find and easy
to load. It seems to me that this would be especially important for a
benchmarking suite, because we would want to encourage people to use
the same suite in Cuis, Pharo, and other images in the Squeak family.

Dave


> 
> We spent some time pulling together benchmarks from various sources
> (papers, the mailinglist, projects on squeaksource, ...) and combining them
> with an extended version of Stefan Marr's implementation of a benchmarking
> framework SMark. The tool and framework are modeled after SUnit, and
> include different execution suites and code to figure out confidence
> variations over multiple runs and such. Also, it draws graphs over multiple
> runs so you can look at things like warmup and GC behavior, and see how
> much time is spent doing incremental GCs and full GCs vs plain execution.
> As a part of this I fixed the EPS export so these graphs can be exported in
> a scalable format.
> 
> Here is a picture of the tool:
> https://dl.dropboxusercontent.com/u/26242153/screenshot.jpg
> 
> As I said, it's modeled after TestRunner and SUnit, benchmarks subclass
> from the "Benchmark" class, any method starting with "bench" is a
> benchmark, and you can have setUp and tearDown methods as usual. By default
> the benchmarks are run under an Autosize runner that re-executes each
> benchmark until the combined runtime reaches 600ms (to smooth out any
> noise). Beyond that, you can specify a number of iterations that the runner
> will re-do that to see multiple averaged runs. The graph shows the
> execution times split between running code (gray) incremental GCs (yellow)
> and full GCs (red). There are popups and you can scroll to zoom in and out.
> There is also a history of benchmark runs stored on the class side of
> benchmark classes for later reference.
> 
> The code currently lives here:
> http://www.hpi.uni-potsdam.de/hirschfeld/squeaksource/BenchmarkRunner
> 
> Considering we are every so often discussing benchmark results here, I
> think it might be useful to share an execution framework for those.
> 
> cheers,
> Tim

> 



More information about the Squeak-dev mailing list