<br><br><div class="gmail_quote">On Tue, Feb 26, 2013 at 3:06 PM, Bert Freudenberg <span dir="ltr"><<a href="mailto:bert@freudenbergs.de" target="_blank">bert@freudenbergs.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
On 2013-02-26, at 21:58, Jeff Gonis <<a href="mailto:jeff.gonis@gmail.com">jeff.gonis@gmail.com</a>> wrote:<br>
<br>
> Hi Everyone,<br>
><br>
> So with a lot of help from Frank Shearar and Nicolas Cellier, I have<br>
> introduced performance benchmarking to the SqueakCI server. You can<br>
> see our current performance trends at the following link:<br>
> <a href="http://build.squeak.org/job/SqueakTrunk/performance/" target="_blank">http://build.squeak.org/job/SqueakTrunk/performance/</a><br>
<br>
</div>Interesting. Why is there already such a large difference between builds #182 and #183? Or are the results simply too noisy? TO be useful they should be fairly consistent.<br>
<div class="im"><br>
> I hope to<br>
> eventually add in benchmarks looking at graphical performance in<br>
> squeak (Balloon vs Gezira vs Cairo, etc), maybe network performance,<br>
> etc, etc. If anyone has any ideas or suggestions I would love to hear<br>
> them.<br>
<br>
<br>
</div>Maybe some macro benchmarks would be useful: opening a browser, finding senders, accepting methods, parsing an xml file, etc.<br></blockquote><div><br></div><div>Be very careful about reflective benchmarks. The speed of finding senders depends on the host hardware, the VM, *and* the number of classes and methods in the system. So any such benchmark results need to be scaled by the number of classes and methods in some way to normalize.</div>
<div><br></div><div>Please *don't* add any such benchmarks to the shootout tests. These are designed to compare the performance of specific algorithms across languages. And one reason I cherry-picked these was to use benchmarks that stressed the VM, not C libraries or plugins. So I'm not particularly interested in adding anything to Shootout that depends on externally-compiled code such as a regexp benchmark that simply tests the compilation of a regexp plugin.</div>
<div><br></div><div>By all means add Squeak-specifc benchmarks, but do so in a different package, please. If you look in Smalltalk-80.sources you'll find the standard set of Smalltalk benchmarks used back in the day (and I expect no one would mind if we stole these). But the find implementors and senders there-in are not yet scaled by the size of the system.</div>
<div><br></div></div>-- <br>best,<div>Eliot</div>