[Box-Admins] Squeak CI Benchmarking Priority

[David T. Lewis]

On Wed, Mar 06, 2013 at 03:21:08PM -0700, Jeff Gonis wrote:
> Hi Everyone,
> 
> So I initially emailed Frank Shearar and he suggested that I send a
> message to this list to get some feedback as well.
> 
> Right now, much to my dismay, the Squeak CI benchmarking is pretty
> useless, showing huge variances in the runtimes of the benchmarks
> between revisions.  I am currently taking a look at the SMark
> framework that Stefan Marr suggested for benchmarking, to see if that
> will help make the benchmarks be of higher quality.
> 
> In the meantime, however, I would like to see what I can do to get
> some quick wins.  On my side I am working on running the benchmarks
> multiple times to warm up the Cog JIT, but I would also like to
> investigate the priority that we run the benchmarks at.
> 
> Right now we launch the vm with the default "nice" priority.  I can't
> get into the box to see what the default is, but I would suspect that
> it is probably 0, as this seems pretty standard. If I bumped the
> priority to -20, to see if this would give the benchmarks a bit more
> stability would anyone be drastically impacted?  They only run when
> the SqueakTrunk project is run, and they shouldn't run for too long.
> I am in fact working on cutting down the runtime a bit to have them
> complete quicker, so we shouldn't have a super high priority process
> running all day every day on the server.
> 
> Does anyone have a large objection to this idea?  Is it ok if I go
> ahead and implement it on a test basis to see if it has an effect on
> the benchmark times?

Hi Jeff,

Raising priority is almost always a bad idea. It will usually not make
anything go faster, and it will probably do horrible things to anything
else happening on the box (such as for example, your login session,
thereby preventing you from further interaction with the system).
Try it if you like, but be prepared to reboot.

Try running top and watching cpu associated with your VM process. If
the CPU is not pretty nearly pegged at 100% of the processor, then
you are probably not CPU bound in the first place, and it's not likely
that raising (or lowering) priority will have much effect.

Given that you are seeing performance variation that seems hard to
explain, I'd be inclined to look at memory utilization first. There
are a lot of things running on the system, including Java stuff that
might well be consuming memory. If the total utilization is such that
the operating system needs to swap pages in and out to keep everyone
happy, this will have a very large impact on the performance of a
Squeak image (including high CPU use). Squeak is very sensitive to
this because it uses a good deal of memory space for the image, and
the operating system has no way of knowing that it should not swap
out pages that are just about to get scanned by the Squeak garbage
collector. If this is happening, then lowering the priority of other
jobs on the system may help (so they get swapped out instead of your
benchmark), but raising the priority of your benchmark job above
normal is still going to be a bad idea.

I'll note also that most of our CI jobs can be run at a lowered
priority in the interest of keeping the system responsive for
interactive use. This will not interfere with the CI jobs and will
make the system appear more responsive even when it is under load.

Dave