Ok, well I think you are into forensic here.  So let me make some suggestions. 

Have a timer based task that wakes up and looks for the  allocated memory > what 200MB? 
When that happens do as a one time shot a count of all the instance allocation sizes, then 
look at dumping all the process stacks so you understand what everything is doing at the time that it wanted & was using 200MB. 

I note the dreadful 1994 defaults for stack dumps are too low, you might need to look at that code that 
creates the stack dump to print more data.  For Sophie we changed things from dumping 4000 bytes to providing upwards of 400Kbytes to help diagnose things. 

If you find that you've a ByteArray that's 150MB then maybe you could override new: for ByteArray and place some 
diagnostics for dumping the caller's stack so you can determine why it's wanting 150MB for a few 100 ms. 


On 2010-10-22, at 12:37 PM, Ken Causey wrote:

> Note that under Linux there are two (really more than two, but two
> major) memory figures.  One is Virtual Set Size (VSZ) and the other is
> Resident Set Size (RSS).  It's nothing new that with Squeak on Linux if
> you don't specify a -memory option it mmaps a gigabyte and this is
> reflected in the VSZ, in my experience this value never changes, I'm not
> sure what would happen if more than a gigabyte was actually addressed.
> I'm not at all concerned about the VSZ value because this does not count
> against true allocated RAM.  So I may be wrong but I believe what you
> are referring to in VSZ and is irrelevant to us.

Yes the VSZ would include the GB mmap, this value is a promise(lie) about what the app could at anytime want. 

> 
> My problem instead is with RSS which represents an actual cost in used
> RAM.  What I am expecting to see is that the RSS value I see is in some
> way representative of the end of memory (vm parameter 3) figure.  This
> is true initially, but as I've said before, when the end of memory drops
> due to a GC, the RSS does not.

If you take a 50MB image and expand it for 100ms to 200MB then RSS will show 200MB, 
If the image vm parameter 3 goes back to 60MB this will NOT be reflected in the RSS value because the 
operating system memory manager gave you the 200MB, this value will decay back to 60MB+  over time 
but that decay is function of some very smart algorithms in the operating system which depend on how much 
RAM other process are using, how much CPU timing Squeak is using. As you know Squeak chews CPU so the 
operating system might pull idle pages from Squeak at a slower rate. 

Frankly here you need to do a "vm_stat 1"  to understand if the operating system is under any paging issues. 
If the RSS is high and the vm parm 3 low, then I'd fully expect the operating system has zero paging going on 
so there is NO effort on it's part to steal pages from the Squeak process because that's expensive and Squeak has 
shown it wanted the pages so why pull them back? 

It's simplistic to consider a high RSS as bad, I'd rather be concerned about seeing a page_in/out rate in vm_stat showing 30+ then I'd bitch... 
Note that at 30 pages/second that's like 1000 ms for hard disk rotation/access/seek,  bad bad, but I guess in new school thought it's only 300ms. 

I'll bet page_in/out rate is zero.. 


> 
> Setting the value of -memory is not going to help here.   That's simply
> going to change the VSZ and set a lower ceiling, a limit on the RSS.
> The reality is that the way the website is emulated there are short
> periods of time in which it needs one or two hundred megabytes of
> memory, but these times are short lived and this is evidence by the end
> of memory figure dropping back to the normal base level.
> 
> Ken

--
===========================================================================
John M. McIntosh <johnmci at smalltalkconsulting.com>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================




-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3829 bytes
Desc: not available
Url : http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20101022/b2ae27f4/smime.bin