Hi Eliot,
On 28 April 2016 at 00:44, Eliot Miranda eliot.miranda@gmail.com wrote:
IMO 600ms is about 500 times too short ;-). Is this parameterised?
This is parameterised, but let me elaborate on why I think this is ok. There are three levels to each benchmark:
1. Problem size. This depends on the benchmark, e.g. a linear progression in BinaryTrees will give you exponentially growing runtime. This should be chosen by the benchmark implementer to be reasonably small while still providing reasonable results. For example, the fibonacci benchmark shouldn't set it's problem size only to 4 or 5, because e.g. on RSqueak the JIT could simply unroll this completely and generate machine code that just checks that the relevant method dicts haven't changed and return the constant. So the problem size should be large enough for that.
2. Autosize iterations These are dynamically chosen per machine to execute a benchmark with a fixed problem size repeatedly to average any noise from e.g. OS-level scheduling and such. I think this is usually fine with 500-1000ms, because OS-level interruptions are then distributed fairly evenly. The autosizer simply finds a small number of iterations to run the inner benchmarks and then averages the runs to get rid of the small noise.
3. Benchmark iterations This is something that you choose when you use the tool, to actually do enough runs to warmup the JIT and such. With an autosize time of about 600ms, I usually choose 100 Benchmark iterations, so that the overall benchmarking time the benchmark runs will be about 60 seconds. In the tool, we measure times and GC stats between these iterations to get the bar chart.
All three of these levels are configurable, but the UI just asks you for the third.
- can it be controlled from the command line?
Yes, provided you mean "use a .st file argument". To run a benchmark you can write e.g. - BenchmarkAutosizeSuite run: {'BenchmarkSimpleStatisticsReporter'. 'SMarkShootout'. 100}. # runs all shootout benchmarks for 100 outer iterations, reporting statistics in the autosize suite - BenchmarkCogSuite run: {'BenchmarkSimpleStatisticsReporter'. 'SMarkShootout.benchBinaryTrees'. 100}. # runs the binarytrees benchmarks for 100 outer iterations without autosizing, but with one extra iteration for warmup ...
Output is printed to stdout.
- is it portable to Pharo?
I don't see why it shouldn't work immediately, unless they don't have the tool builder anymore. Might be that they removed the Postscript Canvas, then you cannot export your benchmarks as easily.
cheers, Tim