squeak gets in this odd state where it consumes lots of cpu according to the operating system tools, but when I do a debug message tally it reports over 90% of the time in idle.
Doing a save always clears up the problem (perhaps because of a global garbage collect?).
Smalltalk vmVersion gives 'Squeak3.8gamma of ''24 November 2004'' [latest update: #6548]'
Running on linux with a largish image (57Mg).
The image itself is a bit old, with #4956 the last update; I think that makes it 3.6 era.
Any ideas what's going on?
On Sun, Jul 03, 2005 at 11:19:37AM -0700, Ross Boylan wrote:
squeak gets in this odd state where it consumes lots of cpu according to the operating system tools, but when I do a debug message tally it reports over 90% of the time in idle.
Let me try to say that more clearly.
Tools running outside of squeak report it is using up most of the CPU on the computer. (basically a top-equivalent).
If I use the debugger inside of squeak, it reports that squeak is mostly idle.
Doing a save always clears up the problem (perhaps because of a global garbage collect?).
I think I can confirm that. I got in this state again, and did Smalltalk garbageCollect That fixed the problem, and answered with the suspiciously large 1029077260 (about 1G, if I'm counting right).
Smalltalk vmVersion gives 'Squeak3.8gamma of ''24 November 2004'' [latest update: #6548]'
Running on linux with a largish image (57Mg).
The image itself is a bit old, with #4956 the last update; I think that makes it 3.6 era.
Any ideas what's going on?
P.S. I think some of the discussion of the website has been unnecessarily heated and personal. If new people get past the site and check out squeak-dev, that kind of stuff is not going to help us.
Ross
Are you swapping? The VM will keep allocating virtual memory space as needed, eventually getting to the point where your OS will start swapping to disk. This might lead to the symptoms you describe.
If you have something like xosview on your system, you will easily see the swap and paging activity if that's what is happening.
Dave
On Mon, Jul 04, 2005 at 09:42:55AM -0700, Ross Boylan wrote:
On Sun, Jul 03, 2005 at 11:19:37AM -0700, Ross Boylan wrote:
squeak gets in this odd state where it consumes lots of cpu according to the operating system tools, but when I do a debug message tally it reports over 90% of the time in idle.
Let me try to say that more clearly.
Tools running outside of squeak report it is using up most of the CPU on the computer. (basically a top-equivalent).
If I use the debugger inside of squeak, it reports that squeak is mostly idle.
Doing a save always clears up the problem (perhaps because of a global garbage collect?).
I think I can confirm that. I got in this state again, and did Smalltalk garbageCollect That fixed the problem, and answered with the suspiciously large 1029077260 (about 1G, if I'm counting right).
Smalltalk vmVersion gives 'Squeak3.8gamma of ''24 November 2004'' [latest update: #6548]'
Running on linux with a largish image (57Mg).
The image itself is a bit old, with #4956 the last update; I think that makes it 3.6 era.
Any ideas what's going on?
On Mon, Jul 04, 2005 at 01:39:54PM -0400, David T. Lewis wrote:
Are you swapping? The VM will keep allocating virtual memory space as needed, eventually getting to the point where your OS will start swapping to disk. This might lead to the symptoms you describe.
If you have something like xosview on your system, you will easily see the swap and paging activity if that's what is happening.
The amount of disk activity seems to be minimal. This is surprising, because I am now seeing the 1G of memory in some of my system tools. First, I don't hear swapping or see the disk access light lit up. Second, when I monitor total disk accesses (using KDE's system guard) they were generally very low (top at 30/sec, usually much lower) when it was sitting there burning CPU, compared to up to 200/s when I saved the image (which again fixed the problem).
$ ps -o rss,size,vsize,%cpu,%mem,command 15215 RSS SZ VSZ %CPU %MEM COMMAND 51012 1053376 1065048 7.2 13.1 squeakvm /ms/d/xfer/squeak/Squeak3.2.image (as noted earlier, it has changesets well past 3.2).
According to my systems docs rss RSS resident set size, the non-swapped physical memory that a task has used (in kiloBytes). (alias rssize, rsz). size SZ approximate amount of swap space that would be required if the process were to dirty all writable pages and then be swapped out. This number is very rough!
vsz VSZ virtual memory size of the process in KiB (1024-byte units). Device mappings are currently excluded; this is subject to change. (alias vsize).
I'm not sure what my page size is, though 1k seems consistent with these numbers.
Note the above high memory numbers accompany low CPU use in the preceding report, so in themselves they aren't enough to cause the problem.
$ free total used free shared buffers cached Mem: 386596 381892 4704 0 46676 69964 -/+ buffers/cache: 265252 121344 Swap: 1267144 791844 475300
On disk, the image is about 40MB, and the changes file 16MB. How the process memory use gets so high (or even if the numbers are to be believed) I don't know.
Dave
On Mon, Jul 04, 2005 at 09:42:55AM -0700, Ross Boylan wrote:
On Sun, Jul 03, 2005 at 11:19:37AM -0700, Ross Boylan wrote:
squeak gets in this odd state where it consumes lots of cpu according to the operating system tools, but when I do a debug message tally it reports over 90% of the time in idle.
Let me try to say that more clearly.
Tools running outside of squeak report it is using up most of the CPU on the computer. (basically a top-equivalent).
If I use the debugger inside of squeak, it reports that squeak is mostly idle.
Doing a save always clears up the problem (perhaps because of a global garbage collect?).
I think I can confirm that. I got in this state again, and did Smalltalk garbageCollect That fixed the problem, and answered with the suspiciously large 1029077260 (about 1G, if I'm counting right).
Smalltalk vmVersion gives 'Squeak3.8gamma of ''24 November 2004'' [latest update: #6548]'
Running on linux with a largish image (57Mg).
The image itself is a bit old, with #4956 the last update; I think that makes it 3.6 era.
Any ideas what's going on?
This could be very well an issue with the remember table holding onto something large which is causing excessive marking activity when the incremental garbage collector runs. If a full GC which you can invoke manually solves the problem, then it most likely is the issue.
To confirm you need to have a linux VM that is built from the latest VMMaker source to pickup my GC Monitor changes, then load the changeset
Visit my Idisk,
http://homepage.mac.com/johnmci/FileSharing.html
GCWork directory
load the JMMGCMonitor.4.cs changeset
Then follow the instructions in
ReadMeForGCStatisticalGathering.rtf
You will find there is logic in GCMonitor>>calculateGoals to force a tenure if the mark count goes greater than 2x the allocation count, usually a sign that the remember table is getting too large, or causing too much marking based on the objects it is looking at in oldspace.
On 4-Jul-05, at 2:33 PM, Ross Boylan wrote:
On Mon, Jul 04, 2005 at 01:39:54PM -0400, David T. Lewis wrote:
Are you swapping? The VM will keep allocating virtual memory space as needed, eventually getting to the point where your OS will start swapping to disk. This might lead to the symptoms you describe.
If you have something like xosview on your system, you will easily see the swap and paging activity if that's what is happening.
The amount of disk activity seems to be minimal. This is surprising, because I am now seeing the 1G of memory in some of my system tools. First, I don't hear swapping or see the disk access light lit up. Second, when I monitor total disk accesses (using KDE's system guard) they were generally very low (top at 30/sec, usually much lower) when it was sitting there burning CPU, compared to up to 200/s when I saved the image (which again fixed the problem).
$ ps -o rss,size,vsize,%cpu,%mem,command 15215 RSS SZ VSZ %CPU %MEM COMMAND 51012 1053376 1065048 7.2 13.1 squeakvm /ms/d/xfer/squeak/ Squeak3.2.image (as noted earlier, it has changesets well past 3.2).
According to my systems docs rss RSS resident set size, the non-swapped physical memory that a task has used (in kiloBytes). (alias rssize, rsz). size SZ approximate amount of swap space that would be required if the process were to dirty all writable pages and then be swapped out. This number is very rough!
vsz VSZ virtual memory size of the process in KiB (1024-byte units). Device mappings are currently excluded; this is subject to change. (alias vsize).
I'm not sure what my page size is, though 1k seems consistent with these numbers.
Note the above high memory numbers accompany low CPU use in the preceding report, so in themselves they aren't enough to cause the problem.
$ free total used free shared buffers cached Mem: 386596 381892 4704 0 46676 69964 -/+ buffers/cache: 265252 121344 Swap: 1267144 791844 475300
On disk, the image is about 40MB, and the changes file 16MB. How the process memory use gets so high (or even if the numbers are to be believed) I don't know.
Dave
On Mon, Jul 04, 2005 at 09:42:55AM -0700, Ross Boylan wrote:
On Sun, Jul 03, 2005 at 11:19:37AM -0700, Ross Boylan wrote:
squeak gets in this odd state where it consumes lots of cpu according to the operating system tools, but when I do a debug message tally it reports over 90% of the time in idle.
Let me try to say that more clearly.
Tools running outside of squeak report it is using up most of the CPU on the computer. (basically a top-equivalent).
If I use the debugger inside of squeak, it reports that squeak is mostly idle.
Doing a save always clears up the problem (perhaps because of a global garbage collect?).
I think I can confirm that. I got in this state again, and did Smalltalk garbageCollect That fixed the problem, and answered with the suspiciously large 1029077260 (about 1G, if I'm counting right).
Smalltalk vmVersion gives 'Squeak3.8gamma of ''24 November 2004'' [latest update: #6548]'
Running on linux with a largish image (57Mg).
The image itself is a bit old, with #4956 the last update; I think that makes it 3.6 era.
Any ideas what's going on?
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On Mon, Jul 04, 2005 at 02:58:47PM -0700, John M McIntosh wrote:
This could be very well an issue with the remember table holding onto something large which is causing excessive marking activity when the incremental garbage collector runs. If a full GC which you can invoke manually solves the problem, then it most likely is the issue.
I'm not very familiar with the GC internals; in particular I've never heard of the "remember table." Is this used to hold onto roots (or maybe everything) in new space that you want to keep?
A characteristic of my image is that has lots of little objects held in a doubly linked list (actually, several doubly linked lists), and I'm always generating more (it's a list of my activities). Does that seem a good prospect for triggering this scenario?
Does "Smalltalk garbageCollect" invoke the global GC? If not, is there a way to do so? The direct invocations I found where in the VM code. As I reported, sometimes "Smalltalk garbageCollect" works, and sometimes it doesn't.
Also, am I correct in inferring from what you say that time spent in the garbage collector is not reported by the debug tallies? There is some reported GC info at the bottom, and it doesn't seem to show much action (copied by hand from report during one of the high CPU/low reported CPU use sessions): **GCs** full 0 totaling 0ms incr 29 totalling 107ms (1.0% uptime), avg 3.0ms tenures 0 root table 0 overflows
To confirm you need to have a linux VM that is built from the latest VMMaker source to pickup my GC Monitor changes, then load the changeset
Is that a prebuilt VM, or do I need to build from sources? (I can't see it right now because I'm downloading over dial up and get a timeout on the page).
Will the fact that I'm on Debian, or the fact that my image is in the 3.6 era, pose any problems with the changeset you mention below?
Visit my Idisk,
http://homepage.mac.com/johnmci/FileSharing.html
GCWork directory
load the JMMGCMonitor.4.cs changeset
Then follow the instructions in
ReadMeForGCStatisticalGathering.rtf
You will find there is logic in GCMonitor>>calculateGoals to force a tenure if the mark count goes greater than 2x the allocation count, usually a sign that the remember table is getting too large, or causing too much marking based on the objects it is looking at in oldspace.
Thanks very much. I'll give it a try.
Also, if anyone has any insights about why my memory footprint is so large compared to the disk files, I'd love to hear it. Perhaps the files saved to disk are compressed?
On Sun, Jul 03, 2005 at 11:19:37AM -0700, Ross Boylan wrote:
squeak gets in this odd state where it consumes lots of cpu according to the operating system tools, but when I do a debug message tally it reports over 90% of the time in idle.
Doing a save always clears up the problem (perhaps because of a global garbage collect?).
I spoke too soon. I just did a Smalltalk garbageCollect and it didn't fix the problem. Saving the image did.
On closer reading I'm no longer sure that Smalltalk garbageCollect always does a full garbage collection, which is implemented in the fullGC primitive.
squeak-dev@lists.squeakfoundation.org