Hi Eliot,

On 28 April 2016 at 00:44, Eliot Miranda <eliot.miranda at gmail.com> wrote:
>
> IMO 600ms is about 500 times too short ;-). Is this parameterised?
>

This is parameterised, but let me elaborate on why I think this is ok.
There are three levels to each benchmark:

1. Problem size.
This depends on the benchmark, e.g. a linear progression in
BinaryTrees will give you exponentially growing runtime. This should
be chosen by the benchmark implementer to be reasonably small while
still providing reasonable results. For example, the fibonacci
benchmark shouldn't set it's problem size only to 4 or 5, because e.g.
on RSqueak the JIT could simply unroll this completely and generate
machine code that just checks that the relevant method dicts haven't
changed and return the constant. So the problem size should be large
enough for that.

2. Autosize iterations
These are dynamically chosen per machine to execute a benchmark with a
fixed problem size repeatedly to average any noise from e.g. OS-level
scheduling and such. I think this is usually fine with 500-1000ms,
because OS-level interruptions are then distributed fairly evenly. The
autosizer simply finds a small number of iterations to run the inner
benchmarks and then averages the runs to get rid of the small noise.

3. Benchmark iterations
This is something that you choose when you use the tool, to actually
do enough runs to warmup the JIT and such. With an autosize time of
about 600ms, I usually choose 100 Benchmark iterations, so that the
overall benchmarking time the benchmark runs will be about 60 seconds.
In the tool, we measure times and GC stats between these iterations to
get the bar chart.

All three of these levels are configurable, but the UI just asks you
for the third.

> - can it be controlled from the command line?

Yes, provided you mean "use a .st file argument". To run a benchmark
you can write e.g.
- BenchmarkAutosizeSuite run: {'BenchmarkSimpleStatisticsReporter'.
'SMarkShootout'. 100}. # runs all shootout benchmarks for 100 outer
iterations, reporting statistics in the autosize suite
- BenchmarkCogSuite run: {'BenchmarkSimpleStatisticsReporter'.
'SMarkShootout.benchBinaryTrees'. 100}. # runs the binarytrees
benchmarks for 100 outer iterations without autosizing, but with one
extra iteration for warmup
...

Output is printed to stdout.


> - is it portable to Pharo?

I don't see why it shouldn't work immediately, unless they don't have
the tool builder anymore. Might be that they removed the Postscript
Canvas, then you cannot export your benchmarks as easily.

>
> cheers,
> Tim
>
>
>
>
>