Here is the Smalltalk stack dump portion of the out of memory messages:

Smalltalk stack dump:
0xbf7d3808 I SmalltalkImage>garbageCollect -1943985568: a(n) SmalltalkImage
0xbf7d3828 I SmalltalkImage>bytesLeft -1943985568: a(n) SmalltalkImage
0xbf7d3850 I SmalltalkImage>lowSpaceWatcher -1943985568: a(n) SmalltalkImage
0x919f60c8 s [] in SmalltalkImage>installLowSpaceWatcher
0x9012dea0 s [] in BlockClosure>newProcess

So it seems the lowSpaceWatcher WAS invoked -- it was just that the
initial GC is the last thing it tried to do, just before it was about
to let my app free a bunch of memory!

In this case, I did use a maximum limit by specifying the -mmap
parameter.  As soon as it hit that, the LowSpace was signaled, it just
couldn't get to asking the first memoryHog to release space because it
couldn't get past the initial GC.

Under these terms, the lowSpaceWatcher is essentially broken and we
should not allow something that important to sit around broken in
Squeak 4.4.  Hence, my thought was along the lines of trying to get
the VM to signal LowSpaceSemaphore just a bit sooner so it would have
enough to do that initial GC -- but I didn't know what you meant by
400K being "quite a lot"..?

 - Chris

(A note about virtual memory):  Every time I install OS these days I
never allocate any Swap anymore because memory is cheap -- you can
still run out of memory+virtual memory so I'd rather things just crash
quickly than go slogging in a swamp and killing my HD before crashing.





On Mon, Mar 11, 2013 at 9:48 PM, David T. Lewis <lewis at mail.msen.com> wrote:
> On Mon, Mar 11, 2013 at 08:51:20PM -0500, Chris Muller wrote:
>> I have a server performing a heavy-lift operation crash with "out of
>> memory".  Investigation revealed the lowSpaceWatcher did not give any
>> of its #memoryHogs a chance to #freeSomeSpace because it the VM
>> actually ran out of memory just prior to that -- during the initial GC
>> (called from #bytesLeft).
>>
>> I notice #lowSpaceThreshold is only 400K which seems like a very
>> insignificant amount of memory by 2013 standards, so I upped it
>> 10-fold to 4M:
>>
>>    lowSpaceThreshold
>>          ^ 4000000
>>
>> My thought was that this would cause the VM to signal the
>> LowSpaceSemaphore, when only 4M was remaining rather than when there
>> was only 400K remaining.  I'm allocating 700M to the server, so a
>> LowSpace signal at 696M is absolutely fine!
>>
>> Unfortunately, the 4M threshold seems to create all sorts of strange
>> slowness and... problems, I guess.
>>
>> I need lowSpaceWatcher to work, what is the proper way to fix this?
>
>
> This is tricky business. The issues collected under this one may give
> some ideas: http://bugs.squeak.org/view.php?id=7655
>
> 400K is actually quite a lot of memory relative to the average size
> of objects that would typically be used in the image. Unless your image
> happens to be allocating some very large objects for some reason, it
> seems more likely that the problem has more to do with waking up the
> low space watcher in time to do something about the problem.
>
> <speculation>
>
> I think that it is also possible, on a unix system, that allowing the
> VM to allocate memory without limit might eventually crash the VM
> with a low space condition as the VM claims more and more virtual
> memory space, gradually leading to swap thrashing and a profoundly
> sluggish image that is unable to schedule anything, including the
> low space watcher process. When you are running Squeak interactively,
> you would notice the sluggish performance and hit the <alt><period>
> interrupt to stop it. But on on a server process, it might just
> bog down and die.
>
> I would suggest, for debugging, to have your server process evaluate
> 'Smalltalk createStackOverflow' while running normally, and observe
> the failure mode. Then try running it again with a fixed memory size
> (using the -memory flag to the VM) in order to force the failure before
> the system starts swapping, and see if the symptoms are different.
>
> </speculation>
>
> Dave
>
>