lowspace signalling and handling issues

[Tim Rowledge]

After having problem trying to debug some TK4 code that blew up with lowspace
problems but never let me catch and debug, I spent some time adding the
lowspace-process stuff we recently discussed. I had to make a few alterations
to match it up with the latest 64bit clean code but no problems with that part.

After building a VM I started testing with some of the methods in
SystemDictionary 'memory space' - in particular  #useUpMemory. It is perhaps
fortunate that I did since the other #useUp* methods pretty much work once the
lowspace process is caught by the vm and passed up to the image. After a _lot_
of head scratching by John & I we found that with a gazillion tiny objects
(Links are the smallest possible objects that can exist on their own, plain
Objects would have to be contained in a collection and so would cost the same 3
words per object) cause a catastrophic GC explosion. What happens is that
memory fills up until we get to signal lowspace and then we are in danger.
Depending upon the exact size of object memory in use the 200kb used as the
lowSpaceThreshold can be gobbled up in one swallow by the
initializeMemoryFirstFree: method making sure there is a byte per object that
survived the markPhase. In using useUpMemory we can get to having 4 bytes of
free space when the next allocate is attempted.... Ka-Boom. 
Expanding the lowSpaceThreshold (along with the VM changes to report the
process and avoid the accidental problem of interrupting eventTickler) to a
couple of mb makes it ok on my machine and the threshold can be a lot lower
with the other tests that create bigger (hence fewer) objects (hence smaller
fwdTable needs). In the worst case, we could have a very large OM filled with
very small objects all surviving markPhase; in such a case we would need an
additional 1/12 of OM available for the fwdTable. So for a 30Mb objectmemory we
ought to set the lowSPaceThreshold to 30/13 => 2.31Mb + actual space needed to
run the notifier/debugger etc for reasonable safety. Or hide  the 2.31 Mb away
so the image never even knows it is there. If you are using virtual memory and
a limit of 512Mb then you should perhaps secrete 40Mb some where safe.

This assumes that we really need to have one byte per object of course. The
original rationale was to keep the number of compact loops down to eight (see
Dan's comment in initializeMemoryFirstFree:) for Alan's large demo image. The
nicest solution would be to come up with a way to do our GC & compacting
without needing any extra space. Commence headscratching now... John suggested
making sure the fwd gets less than the byte-per-object if things are tight, and
accpting the extra compaction loops.

Good news- with the vm change and 2Mb lowSpaceThreshold I can probably go back
and find my TK4 problem(s).

Bad news- consider Tweak. With lots of processes whizzing away, merely stopping
the one that did the allocation and triggered the lowspace is not going to be
much good. Stopping everything except the utterly essential stuff to debug the
lowspace will be needed. Probably.

More bad news- somehow, going from VMM37b5 to b6 cost 40% of performance on my machine :-(  Bugger.

tim
--
Tim Rowledge, tim at sumeru.stanford.edu, http://sumeru.stanford.edu/tim
Every bug you find is the last one.