<div dir="ltr">Hi Tim,<br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 10, 2016 at 12:53 PM, Tim Felgentreff <span dir="ltr"><<a href="mailto:timfelgentreff@gmail.com" target="_blank">timfelgentreff@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br><div dir="ltr">One issue we might want to discuss is the selection and sizing of the benchmarks.<div><br></div><div>All benchmarks are autosized to run at least 600ms right now, and then those 600ms runs are repeated up to 100 times to obtain measurements. But 600ms might be too short for some benchmarks, if the GC only kicks in rarely for them, I don't know.</div></div></blockquote><div><br></div><div>600ms is simply too short. Back in the day when nfib was a popular activation benchmark used to compare different languages (and it's still useful for this today) the approach was to increase the argument to nfib until the result took 30 seconds or more to run. To get activations per second one divided the result of nfib by the time taken.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><br></div><div>About the selection of benchmarks, the ToolInteraction went up, but that is a very high level benchmark, and might also be influenced heavily by refactorings in Morphic. So maybe the benchmark isn't all that useful, or I should stop tracking trunk with the benchmark images, and instead stay on the release.</div></div></div></blockquote><div><br></div><div>The original Smalltalk-80 benchmark suite unclouded senders and implementors whose performance depended not only on the implementation but on the size of the image. I modified the VisualWorks version to scale by the number of methods used. Necessary if one is to measure tool performance, and measuring tool performance is useful in producing a responsive system. But one must be careful to measure something meaningful.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>What do you think?<br></div></div></blockquote><div><br></div><div>Categorising the benchmarks and describing them well is important. Making sure they vary by quality of implementation and not extraneous causes such as code base size, is important. Starting off all benchmarks for a consistent state, and not running a set of benchmarks in the same image (unless one "resets" by throwing away jutted code and GCing) is important. Repeating each benchmark some number of times (e.g. three) and taking the average or the middle, is important. Running benchmarks on an otherwise quiet machine is important.</div><div><br></div><div>And when we have Sista, running the benchmarks such that the system can warm up is important. We will want to see baseline and warm performance compared.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Mon, 10 Oct 2016 at 21:48 Tim Felgentreff <<a href="mailto:timfelgentreff@gmail.com" target="_blank">timfelgentreff@gmail.com</a>> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-4113980806793295631gmail_msg">Indeed, "up" is "slower", but I will think how to add some indication of that the overview page nonetheless. I guess it isn't in there by default, because the speed center website is agnostic to what kinds of benchmarks you use, and writing it on each tiny chart may look messy. But I'll try it out and see how it looks :)<div class="m_-4113980806793295631gmail_msg"><br class="m_-4113980806793295631gmail_msg"></div><div class="m_-4113980806793295631gmail_msg">cheers,</div><div class="m_-4113980806793295631gmail_msg">Tim<br class="m_-4113980806793295631gmail_msg"><div class="m_-4113980806793295631gmail_msg"><br class="m_-4113980806793295631gmail_msg"></div><div class="m_-4113980806793295631gmail_msg"><br class="m_-4113980806793295631gmail_msg"></div></div></div><br class="m_-4113980806793295631gmail_msg"><div class="gmail_quote m_-4113980806793295631gmail_msg"><div dir="ltr" class="m_-4113980806793295631gmail_msg">On Mon, 10 Oct 2016 at 20:26 Bert Freudenberg <<a href="mailto:bert@freudenbergs.de" class="m_-4113980806793295631gmail_msg" target="_blank">bert@freudenbergs.de</a>> wrote:<br class="m_-4113980806793295631gmail_msg"></div><blockquote class="gmail_quote m_-4113980806793295631gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-4113980806793295631gmail_msg"><div class="gmail_extra m_-4113980806793295631gmail_msg"><div class="gmail_quote m_-4113980806793295631gmail_msg">On Mon, Oct 10, 2016 at 7:40 PM, Eliot Miranda <span dir="ltr" class="m_-4113980806793295631gmail_msg"><<a href="mailto:eliot.miranda@gmail.com" class="m_-4113980806793295631gmail_msg" target="_blank">eliot.miranda@gmail.com</a>></span> wrote:<br class="m_-4113980806793295631gmail_msg"></div></div></div><div dir="ltr" class="m_-4113980806793295631gmail_msg"><div class="gmail_extra m_-4113980806793295631gmail_msg"><div class="gmail_quote m_-4113980806793295631gmail_msg"><blockquote class="gmail_quote m_-4113980806793295631gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-4113980806793295631gmail_msg"><div class="gmail_extra m_-4113980806793295631gmail_msg"><div class="gmail_quote m_-4113980806793295631gmail_msg">On Tue, Jul 26, 2016 at 11:56 PM, Tim Felgentreff <span dir="ltr" class="m_-4113980806793295631gmail_msg"><<a href="mailto:timfelgentreff@gmail.com" class="m_-4113980806793295631gmail_msg" target="_blank">timfelgentreff@gmail.com</a>></span> wrote:<br class="m_-4113980806793295631gmail_msg"></div></div></div></blockquote></div></div></div><div dir="ltr" class="m_-4113980806793295631gmail_msg"><div class="gmail_extra m_-4113980806793295631gmail_msg"><div class="gmail_quote m_-4113980806793295631gmail_msg"><blockquote class="gmail_quote m_-4113980806793295631gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-4113980806793295631gmail_msg"><div class="gmail_extra m_-4113980806793295631gmail_msg"><div class="gmail_quote m_-4113980806793295631gmail_msg"><blockquote class="gmail_quote m_-4113980806793295631gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="m_-4113980806793295631gmail_msg">
<a href="http://speed.squeak.org/timeline/#/?exe=1,5&ben=grid&env=2&revs=50&equid=on" rel="noreferrer" class="m_-4113980806793295631gmail_msg" target="_blank">http://speed.squeak.org/<wbr>timeline/#/?exe=1,5&ben=grid&<wbr>env=2&revs=50&equid=on</a></blockquote><div class="m_-4113980806793295631gmail_msg"><br class="m_-4113980806793295631gmail_msg"></div><div class="m_-4113980806793295631gmail_msg">Would it be possible to add so me indication to the page of what is faster? From the graphs I can't tell if higher is faster or longer. There are no labels :-(</div><div class="m_-4113980806793295631gmail_msg"></div></div></div></div></blockquote></div></div></div><div dir="ltr" class="m_-4113980806793295631gmail_msg"><div class="gmail_extra m_-4113980806793295631gmail_msg"><div class="gmail_quote m_-4113980806793295631gmail_msg"><blockquote class="gmail_quote m_-4113980806793295631gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_-4113980806793295631gmail_msg"><div class="gmail_extra m_-4113980806793295631gmail_msg"><div class="gmail_quote m_-4113980806793295631gmail_msg"></div></div></div></blockquote></div><br class="m_-4113980806793295631gmail_msg"></div><div class="gmail_extra m_-4113980806793295631gmail_msg">If you click on one of the tiny charts you get a full chart with labels. In all of the benchmarks, "up" means "more seconds" means "slower".</div></div><div dir="ltr" class="m_-4113980806793295631gmail_msg"><div class="gmail_extra m_-4113980806793295631gmail_msg"><br class="m_-4113980806793295631gmail_msg"></div><div class="gmail_extra m_-4113980806793295631gmail_msg"><div class="m_-4113980806793295631gmail_msg">- Bert -</div></div></div></blockquote></div></blockquote></div>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span style="font-size:small;border-collapse:separate"><div>_,,,^..^,,,_<br></div><div>best, Eliot</div></span></div></div></div>
</div></div>