On Tue, Feb 26, 2013 at 3:06 PM, Bert Freudenberg <bert at freudenbergs.de>wrote:

>
> On 2013-02-26, at 21:58, Jeff Gonis <jeff.gonis at gmail.com> wrote:
>
> > Hi Everyone,
> >
> > So with a lot of help from Frank Shearar and Nicolas Cellier, I have
> > introduced performance benchmarking to the SqueakCI server.  You can
> > see our current performance trends at the following link:
> > http://build.squeak.org/job/SqueakTrunk/performance/
>
> Interesting. Why is there already such a large difference between builds
> #182 and #183? Or are the results simply too noisy? TO be useful they
> should be fairly consistent.
>
> > I hope to
> > eventually add in benchmarks looking at graphical performance in
> > squeak (Balloon vs Gezira vs Cairo, etc), maybe network performance,
> > etc, etc.  If anyone has any ideas or suggestions I would love to hear
> > them.
>
>
> Maybe some macro benchmarks would be useful: opening a browser, finding
> senders, accepting methods, parsing an xml file, etc.
>

Be very careful about reflective benchmarks.  The speed of finding senders
depends on the host hardware, the VM, *and* the number of classes and
methods in the system.  So any such benchmark results need to be scaled by
the number of classes and methods in some way to normalize.

Please *don't* add any such benchmarks to the shootout tests.  These are
designed to compare the performance of specific algorithms across
languages.  And one reason I cherry-picked these was to use benchmarks that
stressed the VM, not C libraries or plugins.  So I'm not particularly
interested in adding anything to Shootout that depends on
externally-compiled code such as a regexp benchmark that simply tests the
compilation of a regexp plugin.

By all means add Squeak-specifc benchmarks, but do so in a different
package, please.  If you look in Smalltalk-80.sources you'll find the
standard set of Smalltalk benchmarks used back in the day (and I expect no
one would mind if we stole these).  But the find implementors and senders
there-in are not yet scaled by the size of the system.

-- 
best,
Eliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20130226/e0544ec8/attachment.htm