I was curious if the VM has any facilities to assist in tracking down memory leaks? (not in the VM itself, but rather in the image it is running) The scenario that comes to mind are object reference loops though there are likely others to consider as well. Any pointers are appreciated.
Thanks, Phil
On Thu, 31 Dec 2015 03:36:46 -0500 "Phil (list)" pbpublist@gmail.com wrote:
I was curious if the VM has any facilities to assist in tracking down memory leaks? (not in the VM itself, but rather in the image it is running) The scenario that comes to mind are object reference loops though there are likely others to consider as well. Any pointers are appreciated.
Hi Phil,
It depends on what, specifically, you are doing.
There are tools in Caregory Tools-ReferenceFinder which I suspect would be of help.
From http://blogten.blogspot.com/
"New Reference Finder / Weight Tracer
During ESUG's Show Us Your Projects, I gave a demo of a really fast reference finder I had written in Cuis a number of years ago. During ESUG, Jan Vrany and I wrote an enhancement to the reference finder: a so-called weight tracer. What this tracer does is tell you which objects only "hang" from a given object. That is, if the given object goes away, the weight tracer shows what other objects would be collected as well.
Initially, the weight tracer required about two minutes per scan. I managed to cut that in half at ESUG with some less than ideal hackery. But today I really understood what was going on. After deleting the unnecessary hackery, and with just a few tiny changes, the weight tracer runs just as quickly as the reference finder: a couple tenths of a second."
==== ====
In the limit, any deep memory bug is probably investigated in a VM simulator. This type of bug in a released image is rare. Typical images do not load the object-memory/vm simulator, but you can build a VMMaker image which has the code.
Note that this is for people with deep knowledge. You have to understand how objects are represented in memory, Cog (JIT) vs non-Cog, Spur vs non-Spur, StackInterpreter vs optimized, ...
Two article blogs discuss the deeper currents within the VM: http://www.mirandabanda.org/cogblog/ https://clementbera.wordpress.com/author/clementbera/
Note that because of the Squeak/Pharo/Cuis community, you can safely ignore the VM and play happily in Smalltalk, so don't feel you have to dive into this level of operations unless you have the calling for it (i.e. you are walking alone and something reaches out from the earth, grabs your ankle, and down you go. Hey, it happens!).
Cheers, -KenD
On Thu, 2015-12-31 at 08:14 -0800, KenD wrote:
On Thu, 31 Dec 2015 03:36:46 -0500 "Phil (list)" pbpublist@gmail.com wrote:
I was curious if the VM has any facilities to assist in tracking down memory leaks? (not in the VM itself, but rather in the image it is running) The scenario that comes to mind are object reference loops though there are likely others to consider as well. Any pointers are appreciated.
Hi Phil,
It depends on what, specifically, you are doing.
There are tools in Caregory Tools-ReferenceFinder which I suspect would be of help.
I've been using reference finder and weight explorer to do some preliminary investigation but given the current size of this image (250M) it's slow going with each search taking upwards of a minute and running into OOM problems. But if that's all there is, I'll keep going down this path.
In the limit, any deep memory bug is probably investigated in a VM simulator. This type of bug in a released image is rare. Typical images do not load the object-memory/vm simulator, but you can build a VMMaker image which has the code.
At this stage I'm expecting to find that it's my code doing the leaking into the image, and not a base image or VM bug. My original thought was that I had some orphaned reference loops floating around but ran across a post that indicated that unrooted reference loops should not be a problem for the VM to gc. I've done all of the obvious things I could think of to minimize rooted references including closing workspaces, browsers, inspectors, terminated Smalltalk processes etc. Any ideas as to what other possible ways these objects could be rooted?
Note that this is for people with deep knowledge. You have to understand how objects are represented in memory, Cog (JIT) vs non- Cog, Spur vs non-Spur, StackInterpreter vs optimized, ...
Hoping not to need to dive this deeply just yet. (esp. for what appears to be a user-space problem)
Two article blogs discuss the deeper currents within the VM: http://www.mirandabanda.org/cogblog/ https://clementbera.wordpress.com/author/clementbera/
Note that because of the Squeak/Pharo/Cuis community, you can safely ignore the VM and play happily in Smalltalk, so don't feel you have to dive into this level of operations unless you have the calling for it (i.e. you are walking alone and something reaches out from the earth, grabs your ankle, and down you go. Hey, it happens!).
I didn't start down this path out of curiosity ;-) You can only ignore it when it doesn't result in your image growing by an order of magnitude larger than it should be and beginning to experience performance issues as a result. I could start from a new base image but as this is code I want to productionize at some point, I'd expect that whatever is going on right now will be a problem down the road so I'd rather deal with it sooner rather than later.
Cheers, -KenD
Phil
Did you try a SpaceTally to know at least, instances of which class are the ones taking most of the space?
On Thu, Dec 31, 2015 at 5:29 PM, Phil (list) pbpublist@gmail.com wrote:
On Thu, 2015-12-31 at 08:14 -0800, KenD wrote:
On Thu, 31 Dec 2015 03:36:46 -0500 "Phil (list)" pbpublist@gmail.com wrote:
I was curious if the VM has any facilities to assist in tracking down memory leaks? (not in the VM itself, but rather in the image it is running) The scenario that comes to mind are object reference loops though there are likely others to consider as well. Any pointers are appreciated.
Hi Phil,
It depends on what, specifically, you are doing.
There are tools in Caregory Tools-ReferenceFinder which I suspect would be of help.
I've been using reference finder and weight explorer to do some preliminary investigation but given the current size of this image (250M) it's slow going with each search taking upwards of a minute and running into OOM problems. But if that's all there is, I'll keep going down this path.
In the limit, any deep memory bug is probably investigated in a VM simulator. This type of bug in a released image is rare. Typical images do not load the object-memory/vm simulator, but you can build a VMMaker image which has the code.
At this stage I'm expecting to find that it's my code doing the leaking into the image, and not a base image or VM bug. My original thought was that I had some orphaned reference loops floating around but ran across a post that indicated that unrooted reference loops should not be a problem for the VM to gc. I've done all of the obvious things I could think of to minimize rooted references including closing workspaces, browsers, inspectors, terminated Smalltalk processes etc. Any ideas as to what other possible ways these objects could be rooted?
Note that this is for people with deep knowledge. You have to understand how objects are represented in memory, Cog (JIT) vs non- Cog, Spur vs non-Spur, StackInterpreter vs optimized, ...
Hoping not to need to dive this deeply just yet. (esp. for what appears to be a user-space problem)
Two article blogs discuss the deeper currents within the VM: http://www.mirandabanda.org/cogblog/ https://clementbera.wordpress.com/author/clementbera/
Note that because of the Squeak/Pharo/Cuis community, you can safely ignore the VM and play happily in Smalltalk, so don't feel you have to dive into this level of operations unless you have the calling for it (i.e. you are walking alone and something reaches out from the earth, grabs your ankle, and down you go. Hey, it happens!).
I didn't start down this path out of curiosity ;-) You can only ignore it when it doesn't result in your image growing by an order of magnitude larger than it should be and beginning to experience performance issues as a result. I could start from a new base image but as this is code I want to productionize at some point, I'd expect that whatever is going on right now will be a problem down the road so I'd rather deal with it sooner rather than later.
Cheers, -KenD
Phil
Mariano,
On Thu, 2015-12-31 at 17:56 -0300, Mariano Martinez Peck wrote:
Did you try a SpaceTally to know at least, instances of which class are the ones taking most of the space?
Chris recently pointed me in that direction. I was previously just looking at the instance counts but looking at memory used quite possibly could lead me in some additional directions.
Thanks, Phil
On Thu, 31 Dec 2015 15:29:02 -0500 "Phil (list)" pbpublist@gmail.com wrote:
At this stage I'm expecting to find that it's my code doing the leaking into the image, and not a base image or VM bug. My original thought was that I had some orphaned reference loops floating around but ran across a post that indicated that unrooted reference loops should not be a problem for the VM to gc.
Making sure such object nets are unrooted as viewed by the gc is usually the problem.
Are you making use of finalization?
Ephemerons in the newer Spur image are useful here -- but it will take some work to get an updated Cuis image in the Spur memory format. Cuis does have a variety of weak collections which can be very helpful.
I've done all of the obvious things I could think of to minimize rooted references including closing workspaces, browsers, inspectors, terminated Smalltalk processes etc. Any ideas as to what other possible ways these objects could be rooted?
I tend to think of expected object lifetimes. Having done real-time systems, one thinks of pre-allocating objects and explicitly managing object lifetimes.
If you have expected lifetimes and a "duty cycle" you should be able to nil out ivars/globals when you reach the expected "end-of-life" to orphan object nets. The run the full collection a couple of times and check that the memory is being reclaimed.
You may be able to gc to get the memory usage, nil out a bunch of references, gc twice again, then check if the particular bunch of references made a difference. Pick a new bunch of references and retry..
Also, you might be able to collect noted references when assigned to get candidates for testing.
Sometimes thinking contrariwise works. You might take a look at the objects saved in a SmartRefStream to see if things are joined in ways you don't expect. You might write a filter which elides expected object references but notes outliers.
What tools could you build to help yourself be successful?
Also you could send me an image and we could Skype/GoogleHangout. I sometimes find describing the problem to someone else in itself causes "aha" to happen.
Good luck, -KenD
On Thu, 2015-12-31 at 13:14 -0800, KenD wrote:
On Thu, 31 Dec 2015 15:29:02 -0500 "Phil (list)" pbpublist@gmail.com wrote:
At this stage I'm expecting to find that it's my code doing the leaking into the image, and not a base image or VM bug. My original thought was that I had some orphaned reference loops floating around but ran across a post that indicated that unrooted reference loops should not be a problem for the VM to gc.
Making sure such object nets are unrooted as viewed by the gc is usually the problem.
Are you making use of finalization?
Not currently. I seem to recall that one issue I ran into was that there were no guarantees as to when finalize would get called and I also remember something about one or more situations where finalize might not get called at all (though I could be misremembering that / thinking of some random Java scenario I ran into.) As a result, when I need finalizers I tend to do it explicitly/manually which is obviously error prone but didn't consider that there might be other less obvious side-effects involved. Aside from situations involving FFI where you might have resources external to the VM involved, are there common cases when failing to finalize a resource (streams perhaps? That's definitely one thing I'm seeing a lot of lying around) might result in objects failing to be gc'd? (I don't doubt it's possible, just trying to understand the scenarios)
Ephemerons in the newer Spur image are useful here -- but it will take some work to get an updated Cuis image in the Spur memory format. Cuis does have a variety of weak collections which can be very helpful.
I use weak collections extensively and am quite curious to play around with ephemerons as soon as Cuis is ready :-)
I've done all of the obvious things I could think of to minimize rooted references including closing workspaces, browsers, inspectors, terminated Smalltalk processes etc. Any ideas as to what other possible ways these objects could be rooted?
I tend to think of expected object lifetimes. Having done real-time systems, one thinks of pre-allocating objects and explicitly managing object lifetimes.
If you have expected lifetimes and a "duty cycle" you should be able to nil out ivars/globals when you reach the expected "end-of-life" to orphan object nets. The run the full collection a couple of times and check that the memory is being reclaimed.
You may be able to gc to get the memory usage, nil out a bunch of references, gc twice again, then check if the particular bunch of references made a difference. Pick a new bunch of references and retry..
Also, you might be able to collect noted references when assigned to get candidates for testing.
Sometimes thinking contrariwise works. You might take a look at the objects saved in a SmartRefStream to see if things are joined in ways you don't expect. You might write a filter which elides expected object references but notes outliers.
All good food for thought. In this particular instance, I literally have what should be an inactive image (i.e. I've shut down and de- referenced everything I can think to and am just trying to understand where/why so many objects are still being held onto)
What tools could you build to help yourself be successful?
I've already got one in mind: looking for object graphs with similar structures. I just need to understand all of the ways in which they can be rooted to go looking for them.
Also you could send me an image and we could Skype/GoogleHangout. I sometimes find describing the problem to someone else in itself causes "aha" to happen.
I appreciate the offer and will take you up on it. I think I have tracked down at least one scenario that you should be able to replicate from a base Cuis image. I'll email the details.
Good luck, -KenD
Thanks, Phil
I suspect some of this text refers to the tools in the Squeak / Pharo image, while some of this text refers to the tools in Cuis. Exactly what tool does the OOM condition occur with?
Andres.
On 12/31/15 12:29 , Phil (list) wrote:
It depends on what, specifically, you are doing.
There are tools in Caregory Tools-ReferenceFinder which I suspect would be of help.
I've been using reference finder and weight explorer to do some preliminary investigation but given the current size of this image (250M) it's slow going with each search taking upwards of a minute and running into OOM problems. But if that's all there is, I'll keep going down this path.
On Fri, 2016-01-01 at 22:25 -0800, Andres Valloud wrote:
I suspect some of this text refers to the tools in the Squeak / Pharo image, while some of this text refers to the tools in Cuis. Exactly what tool does the OOM condition occur with?
Andres.
Yes, in my post I was referring to the Cuis adaptation of the reference finder tool (which I believe was ported from the Squeak version) which is resulting in the OOM. This is not surprising as my starting point from a memory usage standpoint is already fairly high at ~250M. It was this issue (the OOM/slowness) that finally caused me to post my question to this list as I was wondering if the VM might have some facilities to help make tracking this down more efficient.
It was a complete shot in the dark as I wasn't sure if there were any other tools / VM primitives I could leverage for this. The lack of suggestions on this front is causing me to think not and that I'll just have to slog through it with what I have.
On 12/31/15 12:29 , Phil (list) wrote:
It depends on what, specifically, you are doing.
There are tools in Caregory Tools-ReferenceFinder which I suspect would be of help.
I've been using reference finder and weight explorer to do some preliminary investigation but given the current size of this image (250M) it's slow going with each search taking upwards of a minute and running into OOM problems. But if that's all there is, I'll keep going down this path.
Yes, in my post I was referring to the Cuis adaptation of the reference finder tool (which I believe was ported from the Squeak version)
Cuis currently ships a reference finder I wrote, which is not something I ported from Squeak.
I'm not saying Cuis' reference finder will definitely work in your case, but it might help.
Andres.
On Sat, 2016-01-02 at 07:30 -0800, Andres Valloud wrote:
Yes, in my post I was referring to the Cuis adaptation of the reference finder tool (which I believe was ported from the Squeak version)
Cuis currently ships a reference finder I wrote, which is not something I ported from Squeak.
Ahh... I knew you wrote it, but didn't recall that it was originally written using Cuis. It and weight explorer have been quite handy!
I'm not saying Cuis' reference finder will definitely work in your case, but it might help.
Right now, it's the best tool I've found to try to understand what's going on.
Andres.
vm-dev@lists.squeakfoundation.org
-
Andres Valloud -
KenD -
Mariano Martinez Peck -
Phil (list)