On 01.09.2014, at 02:30, Ben Coman btc@openInWorld.com wrote:
Would you leave #bench as it is to avoid invalidating comparisons with previous results, and add some kind of #bench2 ? cheers -ben
I wouldn't think that's necessary. #bench itself is supposed to have a negligible impact on the numbers, so keeping it as low as possible seems appropriate.
There is an argument to be made that if this change impacts the numbers, then we're not measuring anything useful anyway. E.g. cost of the block activation is still in there for each iteration, so maybe it's not worth changing after all:
[3+4] bench ==> '150,000,000 per second.'
[1 to: 150000000 do: [:i | 3 + 4]] timeToRun ==> 386
... which suggests that the block activation has an almost 200% overhead in this case. But that is a fallacy in itself:
[1 to: 150000000 do: [:i | 3 + 4. 3 + 4]] timeToRun ==> 373
... which suggests that the iteration has a 3000% overhead. At least in current Cog, whereas an optimizing JIT might reduce the whole thing into a no-op.
Yes, micro benchmarks are pretty meaningless.
Optimizing #bench does not make them more meaningful, but since it reduces the measurement error, it might still be worth doing?
- Bert -