[Vm-dev] Re: [squeak-dev] What is meant by 'shrink' in the VM parameters?

[Ken Causey]

Thanks for the info John.  We'll take this and watch things for a while.

You are quite probably right about my reading too much into the RSS
value.  I've felt like on the occasions when the RSS does go down that
it is because it is moving into swap.  What has triggered my concern is
when free memory is down to single digit megabytes and swap is half
used.  Looking at the process list the biggest offender that seems to
stick out is the www.squeak.org process.  And so I'm working on it
perhaps worrying about issues of no real concern.  I still need a bit of
convincing though but you have provided information that will help me
decide.

Thank you,

Ken

On Fri, 2010-10-22 at 13:11 -0700, John M McIntosh wrote:
> Ok, well I think you are into forensic here.  So let me make some suggestions. 
> 
> Have a timer based task that wakes up and looks for the  allocated memory > what 200MB? 
> When that happens do as a one time shot a count of all the instance allocation sizes, then 
> look at dumping all the process stacks so you understand what everything is doing at the time that it wanted & was using 200MB. 
> 
> I note the dreadful 1994 defaults for stack dumps are too low, you might need to look at that code that 
> creates the stack dump to print more data.  For Sophie we changed things from dumping 4000 bytes to providing upwards of 400Kbytes to help diagnose things. 
> 
> If you find that you've a ByteArray that's 150MB then maybe you could override new: for ByteArray and place some 
> diagnostics for dumping the caller's stack so you can determine why it's wanting 150MB for a few 100 ms. 
> 
> 
> On 2010-10-22, at 12:37 PM, Ken Causey wrote:
> 
> > Note that under Linux there are two (really more than two, but two
> > major) memory figures.  One is Virtual Set Size (VSZ) and the other is
> > Resident Set Size (RSS).  It's nothing new that with Squeak on Linux if
> > you don't specify a -memory option it mmaps a gigabyte and this is
> > reflected in the VSZ, in my experience this value never changes, I'm not
> > sure what would happen if more than a gigabyte was actually addressed.
> > I'm not at all concerned about the VSZ value because this does not count
> > against true allocated RAM.  So I may be wrong but I believe what you
> > are referring to in VSZ and is irrelevant to us.
> 
> Yes the VSZ would include the GB mmap, this value is a promise(lie) about what the app could at anytime want. 
> 
> > 
> > My problem instead is with RSS which represents an actual cost in used
> > RAM.  What I am expecting to see is that the RSS value I see is in some
> > way representative of the end of memory (vm parameter 3) figure.  This
> > is true initially, but as I've said before, when the end of memory drops
> > due to a GC, the RSS does not.
> 
> If you take a 50MB image and expand it for 100ms to 200MB then RSS will show 200MB, 
> If the image vm parameter 3 goes back to 60MB this will NOT be reflected in the RSS value because the 
> operating system memory manager gave you the 200MB, this value will decay back to 60MB+  over time 
> but that decay is function of some very smart algorithms in the operating system which depend on how much 
> RAM other process are using, how much CPU timing Squeak is using. As you know Squeak chews CPU so the 
> operating system might pull idle pages from Squeak at a slower rate. 
> 
> Frankly here you need to do a "vm_stat 1"  to understand if the operating system is under any paging issues. 
> If the RSS is high and the vm parm 3 low, then I'd fully expect the operating system has zero paging going on 
> so there is NO effort on it's part to steal pages from the Squeak process because that's expensive and Squeak has 
> shown it wanted the pages so why pull them back? 
> 
> It's simplistic to consider a high RSS as bad, I'd rather be concerned about seeing a page_in/out rate in vm_stat showing 30+ then I'd bitch... 
> Note that at 30 pages/second that's like 1000 ms for hard disk rotation/access/seek,  bad bad, but I guess in new school thought it's only 300ms. 
> 
> I'll bet page_in/out rate is zero.. 
> 
> 
> > 
> > Setting the value of -memory is not going to help here.   That's simply
> > going to change the VSZ and set a lower ceiling, a limit on the RSS.
> > The reality is that the way the website is emulated there are short
> > periods of time in which it needs one or two hundred megabytes of
> > memory, but these times are short lived and this is evidence by the end
> > of memory figure dropping back to the normal base level.
> > 
> > Ken
> 
> --
> ===========================================================================
> John M. McIntosh <johnmci at smalltalkconsulting.com>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
> 
> 
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
Url : http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20101022/8aa39cc7/attachment.pgp