On Wed, Mar 06, 2013 at 03:21:08PM -0700, Jeff Gonis wrote:
Hi Everyone,
So I initially emailed Frank Shearar and he suggested that I send a message to this list to get some feedback as well.
Right now, much to my dismay, the Squeak CI benchmarking is pretty useless, showing huge variances in the runtimes of the benchmarks between revisions. I am currently taking a look at the SMark framework that Stefan Marr suggested for benchmarking, to see if that will help make the benchmarks be of higher quality.
In the meantime, however, I would like to see what I can do to get some quick wins. On my side I am working on running the benchmarks multiple times to warm up the Cog JIT, but I would also like to investigate the priority that we run the benchmarks at.
Right now we launch the vm with the default "nice" priority. I can't get into the box to see what the default is, but I would suspect that it is probably 0, as this seems pretty standard. If I bumped the priority to -20, to see if this would give the benchmarks a bit more stability would anyone be drastically impacted? They only run when the SqueakTrunk project is run, and they shouldn't run for too long. I am in fact working on cutting down the runtime a bit to have them complete quicker, so we shouldn't have a super high priority process running all day every day on the server.
Does anyone have a large objection to this idea? Is it ok if I go ahead and implement it on a test basis to see if it has an effect on the benchmark times?
Hi Jeff,
Raising priority is almost always a bad idea. It will usually not make anything go faster, and it will probably do horrible things to anything else happening on the box (such as for example, your login session, thereby preventing you from further interaction with the system). Try it if you like, but be prepared to reboot.
Try running top and watching cpu associated with your VM process. If the CPU is not pretty nearly pegged at 100% of the processor, then you are probably not CPU bound in the first place, and it's not likely that raising (or lowering) priority will have much effect.
Given that you are seeing performance variation that seems hard to explain, I'd be inclined to look at memory utilization first. There are a lot of things running on the system, including Java stuff that might well be consuming memory. If the total utilization is such that the operating system needs to swap pages in and out to keep everyone happy, this will have a very large impact on the performance of a Squeak image (including high CPU use). Squeak is very sensitive to this because it uses a good deal of memory space for the image, and the operating system has no way of knowing that it should not swap out pages that are just about to get scanned by the Squeak garbage collector. If this is happening, then lowering the priority of other jobs on the system may help (so they get swapped out instead of your benchmark), but raising the priority of your benchmark job above normal is still going to be a bad idea.
I'll note also that most of our CI jobs can be run at a lowered priority in the interest of keeping the system responsive for interactive use. This will not interfere with the CI jobs and will make the system appear more responsive even when it is under load.
Dave