<div dir="ltr">+25% is certainly more than noise and a concern<br></div><div class="gmail_extra"><br><div class="gmail_quote">2016-04-03 13:32 GMT+02:00 Holger Freyther <span dir="ltr"><<a href="mailto:holger@freyther.de" target="_blank">holger@freyther.de</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br>Hi Guys,<br>
<br>
for one of our GSM components we have a simple decoding benchmark. The plot of the benchmark runs across Pharo builds can be seen here[1]. One can clearly see the improvement around the 15th of December 2015 when Spur was enabled in Pharo5. Around the 16th January we see a reduction in performance and the values seem to be more "noisy" as well.<br>
<br>
Today I finally had some minutes to try to reproduce this in a more stable environment and just by using Eliots VMs. I have downloaded the last build of 2015 and the four ones available for this year and executed the mini benchmark and wrote the attached script. The results can be seen in results.csv.<br>
<br>
Now is this a "real" regression or something else? E.g. if I decide to not trust the time measurement done by smalltalk itself and use "time" to execute the benchmark, I do see a similar ~500ms difference. Is this of concern?<br>
<br>
If there are more builds I am happy to bisect this further, I can make the benchmark image available as well.<br>
<br>
kind regards<br>
holger<br>
<br>
<br>
<br>
<br><br>
<br>
<br>
<br>
<br>
[1] <a href="https://ci.hartl.name/job/TCAP-pharo5-nightly/plot/" rel="noreferrer" target="_blank">https://ci.hartl.name/job/TCAP-pharo5-nightly/plot/</a><br></blockquote></div><br></div>