<div dir="ltr">Ok, following with this. What I can add to the discussion:<div><br></div><div>In linux, latest VMs yield the following results (I added a space every three digits just to enhance readability)</div><div><br></div>
<div><div>"Pharo Cog"</div><div>1 tinyBenchmarks </div><div>'887 348 353 bytecodes/sec; 141 150 557 sends/sec'</div><div><br></div><div>"Pharo Stack"</div><div>1 tinyBenchmarks </div><div>'445 217 391 bytecodes/sec; 24 395 999 sends/sec'</div>
</div><div><br></div><div>While in Mac</div><div><br></div><div><div>"Pharo Cog"</div><div>1 tinyBenchmarks </div></div><div>'895 104 895 bytecodes/sec; 138 102 772 sends/sec'<br></div><div><br></div><div>
<div>"Pharo Stack"</div><div>1 tinyBenchmarks</div></div><div>'3 319 502 bytecodes/sec; 217 939 sends/sec'<br></div><div><br></div><div><br></div><div>So, I'd say it's a problem in cmake configuration or just compilation in mac :). Though I didn't test on windowze.</div>
<div><br></div><div>Another thing that I noticed is that when compiling my VM on Mac, since I updated Xcode, I was not longer using gnu gcc but llvm one. I tried to go back using the gnu gcc but couldn't make it work so far, he.</div>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Feb 21, 2013 at 5:09 AM, Igor Stasenko <span dir="ltr"><<a href="mailto:siguctua@gmail.com" target="_blank">siguctua@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
On 20 February 2013 18:29, Eliot Miranda <<a href="mailto:eliot.miranda@gmail.com">eliot.miranda@gmail.com</a>> wrote:<br>
><br>
> On Tue, Feb 19, 2013 at 11:10 PM, Camillo Bruni <<a href="mailto:camillobruni@gmail.com">camillobruni@gmail.com</a>> wrote:<br>
>><br>
>><br>
>> On 2013-02-20, at 01:25, Eliot Miranda <<a href="mailto:eliot.miranda@gmail.com">eliot.miranda@gmail.com</a>> wrote:<br>
>><br>
>>><br>
>>> On Tue, Feb 19, 2013 at 2:16 PM, Camillo Bruni <<a href="mailto:camillobruni@gmail.com">camillobruni@gmail.com</a>> wrote:<br>
>>>><br>
>>>>>> The most annoying piece is Time machine and its disk access, I<br>
>>>>>> sometimes forget to suspend it, but it was off during the<br>
>>>>>> tinyBenchmark.<br>
>>>>><br>
>>>>> One simple approach is to run the benchmark three times and to discard<br>
>>>>> the best and the worst results.<br>
>>>><br>
>>>> that is as good as taking the first one... if you want decent results<br>
>>>> measure >30 times and do the only scientific correct thing: avg + std deviation?<br>
>>><br>
>>> If the benchmark takes very little time to run and you're trying to<br>
>>> avoid background effects then your approach won't necessarily work<br>
>>> either.<br>
>><br>
>> true, but the deviation will most probably give you exactly that feedback.<br>
>> if you increase the runs but the quality of the result doesn't improve<br>
>> you know that you're dealing with some systematic error source.<br>
>><br>
>> This approach is simply more scientific and less home-brewed.<br>
><br>
> Of course, no argument here. But what's being discussed is using<br>
> tinyBenchmarks as a quick smoke test. A proper CI system can be set<br>
> it up for reliable results, but for IMO for a quick smoke test doing<br>
> three runs manually is fine. IME, what tends to happen is that the<br>
> first run is slow (caches heating up etc) and the second two runs are<br>
> extremely close.<br>
<br>
</div></div>but not in case when you have an order(s) of magnitude speed<br>
degradation. This is too significant to be<br>
considered as measurement error or deviation.<br>
There should be something wrong with VM (cache always fails?).<br>
<span class="HOEnZb"><font color="#888888"><br>
> --<br>
> best,<br>
> Eliot<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
--<br>
Best regards,<br>
Igor Stasenko.<br>
</div></div></blockquote></div><br></div>