Webteam October 2010

webteam@lists.squeakfoundation.org

1 participants
5 discussions

Re: [squeak-dev] What are we leaking in the www.squeak.org process?
by Ken Causey 21 Oct '10

21 Oct '10

Janko and I have been doing some fiddling around and monitoring today and I begin to develop a theory. First, meet some images: Image 1: SqueakMap (map.squeak.org). This is our oldest Squeak service image with the possible exception of wiki.squeak.org. This is a 3.8 image running on a 3.8 VM. SqueakMap saves it's data to a separate file and snapshots of the image are only made when source code changes occur, aka manually. Image 2: SqueakSource (source.squeak.org). This is a 3.11 image running on a 3.11 VM. This image saves it's data by snapshoting hourly. Image 3: www.squeak.org. This is a 3.9 VM that has been running for some time on the same 3.8 VM as SqueakMap but we just a short while ago decided to try it on the 3.11 VM SqueakSource is using and it is running fine so far. So for a few hours now I have been monitoring the memory usage and GC behavior of these three images. The short story is that SqueakMap is almost completely rock solid but that the SqueakSource image, despite my earlier claims to the contrary, appears to suffer a much reduced form of the problem we are observing on www.squeak.org. To reiterate: The problem is that the resident set size of the process appears to continually grow. The operating system will occasionally swap out some of this giving the impression that the RSS has dropped but I'm almost certain that the total cost (RAM+swap) of the process is the same and continuing to increase. Our available RAM and swap on the server is quite limited with all the various services we are running. What I'm observing is a discrepancy between what the VM claims is the end of memory and the RSS. When any of these images start this discrepancy is quite small, maybe a megabyte. But whenever a GC occurs the end of memory figure drops, but the RSS goes up. So from the images side it all appears copacetic: we cleaned up the garbage and we are more or less back at our base memory usage level. But as far as the operating system is concerned nothing has happened, in fact more memory has been claimed. So my theory is that somehow when GC occurs on the linux VM (3.8 VM and 3.11 VM) the freed up memory is not released to the operating system but it seems that new objects go in a different memory location from the operating system's perspective therefore using additional memory on top of what was used before the GC. I only really observe this on full GCs. When few if any full GCs occur, there is no real problem. But when you do things like snapshot the image hourly, this all adds up to several megabytes a day of lost RAM. This may not be quite right, I'm still observing and of course this theory is built out of black box observation and little else. Feel free to set me straight. Ken

1 0

RE: [Webteam] Re: [squeak-dev] What are we leaking in the www.squeak.org process?
by Ken Causey 20 Oct '10

20 Oct '10

On Thu, 2010-10-14 at 19:00 -0700, Ken Causey wrote: > > -------- Original Message -------- > > Subject: Re: [Webteam] Re: [squeak-dev] What are we leaking in the > > www.squeak.org process? > > From: Igor Stasenko <siguctua(a)gmail.com> > > Date: Thu, October 14, 2010 8:45 pm > > To: The general-purpose Squeak developers list > > <squeak-dev(a)lists.squeakfoundation.org> > > > > > > On 15 October 2010 04:10, Ken Causey <ken(a)kencausey.com> wrote: > > > Another update with more data conflicting somewhat with my last report: > > > > > > Shown by ps: 214560KB RSS > > > Shown by vmParameters (2,3): 64.6MB/67.4MB > > > > > > A discrepancy of some 150MB. > > > > My understanding is, that Squeak VM cannot shrink a memory which were > > commited (due to peak growth), > > to give it back to OS. > > > > So, if at some peak, VM comitted 150 MB, and OS sees it as 150M, > > while VM after GC actually using just 64M. > > No, I don't believe this is true. The RSS does go both down as well as > up. I thought I made that clear in my original comments. > > Although I suppose it is possible that drops in the RSS figure are due > to paging out to swap. I'm not sure how to really check that. > > Ken For example, another data point: Shown by ps: 167344KB RSS Shown by vmParameters (2,3): 62.9MB/65.8MB The process has not been restarted but the RSS has dropped. Ken > > > > > > > > Ken > > > > > > On Thu, 2010-10-14 at 17:31 -0500, Ken Causey wrote: > > >> An update as things may not be quite as I originally described them. It > > >> seems the values reported by vmParameters 2 and 3 are not as constant as > > >> I had been lead to believe. I noticed that just a short while after > > >> sending the image the RSS had jumped up to 165800KB. I still had VNC > > >> open on the image so I checked what is shown there and I see > > >> 145.6MB/147.7MB so there is still a discrepancy, but not perhaps as wide > > >> as I indicated. > > >> > > >> I also did not mention that in addition to restarting Swazoo every 15 > > >> minutes there is other cleanup that is done once a day related to > > >> statistics gathering that goes on. My understanding is that this > > >> cleanup is meant to return things very close to the starting state. > > >> Perhaps Janko can comment on that more fully. > > >> > > >> Ken > > >> > > >> On Thu, 2010-10-14 at 16:55 -0500, Ken Causey wrote: > > >> > For some time now I've been bugging Janko about the memory usage of the > > >> > process that serves http://www.squeak.org/ . I've gathered a bit more > > >> > info today and I'd like to see if anyone here has any thoughts. > > >> > > > >> > First while there is some variation most of the Squeak processes that > > >> > serve squeak.org sites (source.squeak.org, map.squeak.org, > > >> > wiki.squeak.org) have a more or less constant and reasonable memory > > >> > usage. Let me clarify that what I am talking about here is RSS > > >> > (Resident Set Size) as reported by ps/top on Linux. Swiki as the oldest > > >> > of the group is the clear winner at just under 30MB, squeakmap is around > > >> > 47MB, and source.squeak.org is around 59MB. > > >> > > > >> > In contrast immediately after starting it www.squeak.org is using about > > >> > 70MB and it grows from there. Now a bit more than an hour since the > > >> > last restart it is already up to about 85MB. I have seen it over 300MB > > >> > before, and it commonly uses 140-250MB. It doesn't simply grow and > > >> > grow, it does shrink occasionally. > > >> > > > >> > First, there is a known issue with Swazoo that it does not close sockets > > >> > correctly and Janko has addressed this by having Swazoo automatically > > >> > restarted every 15 minutes. So using netstat for example I can watch > > >> > the socket count for the process continually grow and then it drops to > > >> > zero and this repeats. In contrast for all the other process (all > > >> > running some variation of Kom/Comanche I believe) I'm hard pressed to > > >> > even catch a single open socket at any time (not counting the listener > > >> > that is). > > >> > > > >> > Secondly, since I've been complaining about this issue for a long time > > >> > Janko has running a little morph that shows current memory usage. What > > >> > this shows is the values of SmalltalkImage current vmParameterAt: 2 and > > >> > 3. These numbers are quite stable. For example at this moment these > > >> > are 68.3MB and 71.0MB respectively. Whereas ps shows the RSS at > > >> > 85752KB. Where is the discrepancy here? > > >> > > > >> > My current theory is that the vmParameters are not reflecting incidental > > >> > memory use by the VM and plugins or that we are simply misinterpreting > > >> > the values. Is it possible that Swazoo is failing to release something > > >> > that is causing the VM to use more memory than it should? > > >> > > > >> > Note that the same VM is used by all of these processes. It is > > >> > > > >> > 3.8a-1 #1 Sun May 1 19:46:46 EDT 2005 gcc 3.3.5 > > >> > > > >> > Yes, I know it's old, but 'if it ain't broke, don't fix it' has been the > > >> > motto and with the possible exception of Swazoo/www.squeak.org it has > > >> > been working wonderfully for a long time now. > > >> > > > >> > Your thoughts would be greatly appreciated, > > >> > > > >> > Ken > > >> > > >> _______________________________________________ > > >> Webteam mailing list > > >> Webteam(a)lists.squeakfoundation.org > > >> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/webteam > > > > > > > > > > > > > > > > > > > > > > > -- > > Best regards, > > Igor Stasenko AKA sig. > > >

1 1

Fixed: "Error: Attempt to evaluate a block that is already being evaluated"
by Ken Causey 16 Oct '10

16 Oct '10

Janko, That error "Error: Attempt to evaluate a block that is already being evaluated" came up again and so I decided to take a look. I'm almost certain the cause is that you defined a #startUp method on SpEnvironment class that is meant to be executed whenever the image starts up. This is fine and good except that the way this mechanism works a message #startUp: with a boolean argument indicating whether or not this is a true startup (aka resuming) indicated by true, or whether this is simply after a snapshot is made, indicated by false, is sent. Sort of odd I guess but the same mechanism is used for quitting and snapshotting. Behavior defines #startUp: to simply send #startUp and so your existing implementation worked but it was also being called after a snapshot and not a true restart. So I defined #startUp: on SpEnvironment class that only takes the action if the argument is true. I left the old #startUp for now, though it should no longer be used and I left a browser open on the added method so you can look at it and modify as appropriate if you like. I did not update any packages so you will want to take care of that also. Ken

1 0

What are we leaking in the www.squeak.org process?
by Ken Causey 15 Oct '10

15 Oct '10

For some time now I've been bugging Janko about the memory usage of the process that serves http://www.squeak.org/ . I've gathered a bit more info today and I'd like to see if anyone here has any thoughts. First while there is some variation most of the Squeak processes that serve squeak.org sites (source.squeak.org, map.squeak.org, wiki.squeak.org) have a more or less constant and reasonable memory usage. Let me clarify that what I am talking about here is RSS (Resident Set Size) as reported by ps/top on Linux. Swiki as the oldest of the group is the clear winner at just under 30MB, squeakmap is around 47MB, and source.squeak.org is around 59MB. In contrast immediately after starting it www.squeak.org is using about 70MB and it grows from there. Now a bit more than an hour since the last restart it is already up to about 85MB. I have seen it over 300MB before, and it commonly uses 140-250MB. It doesn't simply grow and grow, it does shrink occasionally. First, there is a known issue with Swazoo that it does not close sockets correctly and Janko has addressed this by having Swazoo automatically restarted every 15 minutes. So using netstat for example I can watch the socket count for the process continually grow and then it drops to zero and this repeats. In contrast for all the other process (all running some variation of Kom/Comanche I believe) I'm hard pressed to even catch a single open socket at any time (not counting the listener that is). Secondly, since I've been complaining about this issue for a long time Janko has running a little morph that shows current memory usage. What this shows is the values of SmalltalkImage current vmParameterAt: 2 and 3. These numbers are quite stable. For example at this moment these are 68.3MB and 71.0MB respectively. Whereas ps shows the RSS at 85752KB. Where is the discrepancy here? My current theory is that the vmParameters are not reflecting incidental memory use by the VM and plugins or that we are simply misinterpreting the values. Is it possible that Swazoo is failing to release something that is causing the VM to use more memory than it should? Note that the same VM is used by all of these processes. It is 3.8a-1 #1 Sun May 1 19:46:46 EDT 2005 gcc 3.3.5 Yes, I know it's old, but 'if it ain't broke, don't fix it' has been the motto and with the possible exception of Swazoo/www.squeak.org it has been working wonderfully for a long time now. Your thoughts would be greatly appreciated, Ken

1 2

www.squeak.org down
by Ken Causey 14 Oct '10

14 Oct '10

A quick note to let you know that the www.squeak.org process is not responding currently. It is pegging the CPU. Ken

1 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Webteam October 2010