Hi Everyone,
So I initially emailed Frank Shearar and he suggested that I send a message to this list to get some feedback as well.
Right now, much to my dismay, the Squeak CI benchmarking is pretty useless, showing huge variances in the runtimes of the benchmarks between revisions. I am currently taking a look at the SMark framework that Stefan Marr suggested for benchmarking, to see if that will help make the benchmarks be of higher quality.
In the meantime, however, I would like to see what I can do to get some quick wins. On my side I am working on running the benchmarks multiple times to warm up the Cog JIT, but I would also like to investigate the priority that we run the benchmarks at.
Right now we launch the vm with the default "nice" priority. I can't get into the box to see what the default is, but I would suspect that it is probably 0, as this seems pretty standard. If I bumped the priority to -20, to see if this would give the benchmarks a bit more stability would anyone be drastically impacted? They only run when the SqueakTrunk project is run, and they shouldn't run for too long. I am in fact working on cutting down the runtime a bit to have them complete quicker, so we shouldn't have a super high priority process running all day every day on the server.
Does anyone have a large objection to this idea? Is it ok if I go ahead and implement it on a test basis to see if it has an effect on the benchmark times?
Thanks for your feedback, Jeff