[Vm-dev] Memory leaks

[Phil (list)]

On Thu, 2015-12-31 at 13:14 -0800, KenD wrote:
> On Thu, 31 Dec 2015 15:29:02 -0500
> "Phil (list)" <pbpublist at gmail.com> wrote:
> 
> > At this stage I'm expecting to find that it's my code doing the
> > leaking
> > into the image, and not a base image or VM bug.  My original
> > thought
> > was that I had some orphaned reference loops floating around but
> > ran
> > across a post that indicated that unrooted reference loops should
> > not
> > be a problem for the VM to gc. 
> 
> Making sure such object nets are unrooted as viewed by the gc is
> usually the problem.
> 
> Are you making use of finalization?

Not currently.  I seem to recall that one issue I ran into was that
there were no guarantees as to when finalize would get called and I
also remember something about one or more situations where finalize
might not get called at all (though I could be misremembering that /
thinking of some random Java scenario I ran into.)  As a result, when I
need finalizers I tend to do it explicitly/manually which is obviously
error prone but didn't consider that there might be other less obvious
side-effects involved.  Aside from situations involving FFI where you
might have resources external to the VM involved, are there common
cases when failing to finalize a resource (streams perhaps?  That's
definitely one thing I'm seeing a lot of lying around) might result in
objects failing to be gc'd? (I don't doubt it's possible, just trying
to understand the scenarios)

> 
> Ephemerons in the newer Spur image are useful here -- but it will
> take some work to get an updated Cuis image in the Spur memory
> format.   Cuis does have a variety of weak collections which can be
> very helpful.
> 

I use weak collections extensively and am quite curious to play around
with ephemerons as soon as Cuis is ready :-)

> >  I've done all of the obvious things I
> > could think of to minimize rooted references including closing
> > workspaces, browsers, inspectors, terminated Smalltalk processes
> > etc.
> >  Any ideas as to what other possible ways these objects could be
> > rooted?
> 
> I tend to think of expected object lifetimes.  Having done real-time
> systems, one thinks of pre-allocating objects and explicitly managing
> object lifetimes.
> 
> If you have expected lifetimes and a "duty cycle" you should be able
> to nil out ivars/globals when you reach the expected "end-of-life" to
> orphan object nets.  The run the full collection a couple of times
> and check that the memory is being reclaimed.
> 
> You may be able to gc to get the memory usage, nil out a bunch of
> references, gc twice again, then check if the particular bunch of
> references made a difference.  Pick a new bunch of references and
> retry..
> 
> Also, you might be able to collect noted references when assigned to
> get candidates for testing.  
> 
> Sometimes thinking contrariwise works.  You might take a look at the
> objects saved in a SmartRefStream to see if things are joined in ways
> you don't expect.  You might write a filter which elides expected
> object references but notes outliers.
> 

All good food for thought.  In this particular instance, I literally
have what should be an inactive image (i.e. I've shut down and de-
referenced everything I can think to and am just trying to understand
where/why so many objects are still being held onto)

> What tools could you build to help yourself be successful?
> 

I've already got one in mind: looking for object graphs with similar
structures.  I just need to understand all of the ways in which they
can be rooted to go looking for them.

> Also you could send me an image and we could Skype/GoogleHangout.  I
> sometimes find describing the problem to someone else in itself
> causes "aha" to happen.  
> 

I appreciate the offer and will take you up on it.  I think I have
tracked down at least one scenario that you should be able to replicate
from a base Cuis image.  I'll email the details.

> Good luck,
> -KenD

Thanks,
Phil