<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">Hi Chris,<div dir="ltr"></div><div dir="ltr"><br><blockquote type="cite">On Dec 17, 2019, at 4:36 PM, Chris Muller <asqueaker@gmail.com> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div>Benchmarking is ultimately about finding what performs best, and this invariably involves a lot of manual benching, recording, and comparing of different individual expressions. I've been wanting to optimize this process for a long time, and today I finally took the time to spank out a first draft of this new feature in Chronology-Core-cmm.52.</div><div><br></div><div>Here's an example inspired by my discovery about creating Dictionary's with a pre-allocated size:</div><div><br></div><div> { [Dictionary new].</div> [Dictionary new: 4]} benchCompare <div><br></div><div>answers an Array with corresponding measurements, along with an <font color="#cc0000">extra field</font> at the beginning:<br><div><br></div><div> {'[Dictionary new]'->'<font color="#cc0000">100% of baseline rate</font>, 26,300,000 per second. 38.1 nanoseconds per run. 7.85843 % GC time.' . </div> '[Dictionary new: 4]'->'<font color="#cc0000">13% of baseline rate</font>, 3,510,000 per second. 285 nanoseconds per run. 1.18 % GC time.'}"</div><div><br></div><div>The first one is the baseline, and so will always report "100%". The subsequent ones are relative to that baseline.</div><div><br></div><div>Since this is a new feature, it won't make it into 5.3, but I would like put this into trunk after the release. Please let me know your thoughts.</div></div></div></blockquote><div><br></div>The two key things with something like this above our current vm technology (especially now we’re moving to full blocks) are<div>- evaluate each block a few times (if it is short running) to ensure it has been jitted</div><div>- subtract the cost of a simple block. For example in the above [Dictionary] is a useful null block that costs to evaluate the block and to indirect through the #Dictionary-> Dictionary association.</div><div><br></div><div>So if you changed your scheme to <br><div><div dir="ltr"><div dir="ltr"><div><br></div><div> { [Dictionary new].</div> [Dictionary new: 4]} benchCompareWithNullCase: [Dictionary]</div><div dir="ltr"><br></div><div dir="ltr">you could get more meaningful results.</div><div dir="ltr"><br></div><div dir="ltr">There are other repeatability concerns; running the scavenger before each individual case is measured to ensure they run in a more fair way with an empty eden, or even running a full GC. These are useful parameyerisations. So eg</div><div dir="ltr"><br></div></div><div><div> { [Dictionary new].</div> [Dictionary new: 4]}</div><div> benchCompareWithNullCase: [Dictionary]</div><div> beforeEachDo: [Smalltalk garbageCollectMost]</div><div><br></div><div>and automatically compiling a benchmark block with more repetitions, to avoid results getting swamped by block evaluation overhead (an issue for micro benchmarks). So eg if any element of the initial array was an association from block to integer the method would construct a block with that many repetitions (which could be gone textually or by bytecode magic) and measure that instead.</div><br><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div><br></div><div>Best,</div><div> Chris</div></div>
<span></span><br></div></blockquote></div></div></body></html>