On Wed, 2010-10-20 at 00:16 -0700, Colin Putney wrote:
> On Tue, Oct 19, 2010 at 10:35 PM, Ken Causey <ken at kencausey.com> wrote:
> 
> 
> > I would really really appreciate some guidance here.
> 
> I'd assume that the VM is working fine, and something in the image is
> what's using the memory. Try looking at counts of instances by class
> and see if anything sticks out, then get a hold of one of them and
> trace the references back and see where it's being referenced from.
> Look at running Processes and see if there are any with odd things on
> the stack that could reference a large tree of objects. Look at
> Morphic: explore the World, the deferred messages queue and the ui
> process.
> 
> Also look at the VM process from outside, via the OS. How many file
> handles does it have open? What files, sockets, pipes etc do they
> refer to? What shared libraries are loaded? Can you profile it and see
> where it's spending most of its processing time? How do all these
> things differ between a VM process that's just started, and one that's
> been running a while and is using a lot of memory?
> 
> Just a few suggestions. Good luck!
> 
> Colin

Thanks Colin, but it seems I was not clear.  This is an issue that has
been going on for a long time now and the low hanging fruit has already
been picked.  Janko Mivsek has investigated to the best of his ability
and dooesn't believe there is any garbage that is not being dereferenced
and not collected.  This is very strongly reinforced by the usage of a
morphic tool call 'Memory Usage' which shows the values of VM parameters
2 and 3.  With the exception of short lived peaks, the values shown here
are stable in the range of 60-70MB at all times for weeks and months of
time.

I don't have any reason to think there is a problem with the VM, it is
the same VM most of the Squeak processes have been using on our server
for some time now.  But the rest all have very consistent solid memory
usage from the operating system perspective.  This is not the case for
the website process which is using the very same VM.  So I agree that
something in the image is probably wrong, or perhaps Janko has succeeded
in tickling a bug that the others have not.

Regarding looking at the VM process from outside.  Please read my long
message more carefully, I realize it's not formatted terribly well,
sorry about that.  But I mention another issue which is known regarding
failures to properly close sockets.  But as mentioned this is cleared up
every 15 minutes, which I can see from the operating system side.  But
this cleanup is not reflected in any way in the memory usage.  This is a
difference compared to other Squeak processes we are running and maybe
the problem is related to this, but it's non-obvious and help is needed
to understand why, if that's the case.

I'm no lsof expert, but a quick look at the output there doesn't
indicate anything surprising to me, certainly no great number of files
or other file-like items open.

Near the end of the message you seem to imply some sort of issue with
the problem getting worse over time.  Apparently I was less than clear
on this.  The problem with the memory issue can happen within a matter
of minutes of startup, certainly within an hour or two.  And it doesn't
get particularly worse with time.

The fundamental issue is simply that for no good reason that we have
been able to discover the website process uses several times as much
memory as any other of our Squeak-based services.

At this point the thing that clearly differs for this process is the
discrepancy between what the VM reports regarding memory usage and that
reported by the operating system.  The amount of this discrepancy also
varies seemingly in step with the RSS reported by the operating system
in contrast to the reported memory usage by the VM which is
comparatively constant.

Ken
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
Url : http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20101020/f922c3f7/attachment.pgp