So, recent versions of OSProcess have this neat trick of being able to fork and save the image in a background process, which means you can have a large long-running image - say, a Seaside server - drop a backup of itself to disk without, hopefully, skipping a beat. In theory, because of the copy-on-write forking that modern unices use, it should even be able to do this without allocating much extra memory as long as it does it quickly enough. However, since every snapshot is preceded by a full GC, which moves tons of memory around, in practice what I see is that memory usage doubles whenever I do the background save.
How necessary is it to do the GC before a snapshot? Is it possible to do a writeImageFile() without a GC and safely use the resulting image, or does that violate some assumptions somewhere?
Avi
So, recent versions of OSProcess have this neat trick of being able to fork and save the image in a background process, which means you can have a large long-running image - say, a Seaside server - drop a backup of itself to disk without, hopefully, skipping a beat. In theory, because of the copy-on-write forking that modern unices use, it should even be able to do this without allocating much extra memory as long as it does it quickly enough. However, since every snapshot is preceded by a full GC, which moves tons of memory around, in practice what I see is that memory usage doubles whenever I do the background save.
How necessary is it to do the GC before a snapshot? Is it possible to do a writeImageFile() without a GC and safely use the resulting image, or does that violate some assumptions somewhere?
Avi
On 10/6/05, Avi Bryant avi.bryant@gmail.com wrote:
How necessary is it to do the GC before a snapshot?
I doubt that anybody knows. If I had been the author of the snapshotting code, I would have included a GC from the beginning, and perhaps nobody since then would have had a reason to take it out. (If you ever have to look at an image file's internals for debugging reasons, it will be "cleaner" after a GC, for one thing.)
Is it possible to do a writeImageFile() without a GC and safely use the resulting image, or does that violate some assumptions somewhere?
An image before a GC should be just as valid as the same image after a GC. If you find otherwise, it's a bug. Nevertheless, that will be of little comfort if you find yourself struggling to reconstruct a working image on a deadline.
My advice, which probably says nothing you didn't already know, is to make a backup, carefully eyeball the snapshotting code, then give your idea a try, with lots of testing before you commit to it. I don't see why it wouldn't work, although moving the GC to happen before the fork would perhaps solve your memory thrashing problems with fewer worries.
--Tom Phoenix
Hi Avi--
I'm pretty sure you don't need to GC before making a snapshot. I've done this when making snapshots from the simulator, and haven't run into any untoward behavior after resuming.
I think it's just to minimize the size of the snapshot on disk.
Of course, the person to answer this is Dan.
-C
You might get away with no GC before a snapshot but please remember that you do need to do various shutdown work to make the written snapshot have a serious chance of being restartable.
I'm not in a position to provide a list; never tried it so I don't know what would go wrong. I _do_ know that a similar 'neat trick' cost PPS a very large amount of engineer time (and therefore direct money out of my budget as manager of the dept. back then) when so twerp at a Large Oval Logo Company decided that snapshotting in the background every few minutes would be a Good Idea. I can't recall the details any longer - getting too old - but it took _months_ to track down the problem. It was escalated to a total emergency, the GC was blamed, my engineers were blamed, I was blamed, the customer was adamant that they had never, would never, could never have done anything that could have caused it. When we solved the problem and proved it was their code I don't recall anybody even saying thanks.
So all in all I'm not sure I'd recommend doing a regular raw snapshot in the background.
tim
You can *not* get away without a GC, simply because the GC state isn't preserved across snapshots. We actually had some obscure crashes related to this problem in the past (anyone remember the crashes caused by the root bit being set on the active context after a snapshot?)
In other words, the full GC is required to save the image in a well-known "right-after-full-gc-state". One would (at least) have to save sufficient information for the garbage collector to make this work. Likely other fixes too, since a full garbage collect cleans up many, many things that we usually don't even think about (cached contexts for example).
Cheers, - Andreas
tim Rowledge wrote:
You might get away with no GC before a snapshot but please remember that you do need to do various shutdown work to make the written snapshot have a serious chance of being restartable.
I'm not in a position to provide a list; never tried it so I don't know what would go wrong. I _do_ know that a similar 'neat trick' cost PPS a very large amount of engineer time (and therefore direct money out of my budget as manager of the dept. back then) when so twerp at a Large Oval Logo Company decided that snapshotting in the background every few minutes would be a Good Idea. I can't recall the details any longer - getting too old - but it took _months_ to track down the problem. It was escalated to a total emergency, the GC was blamed, my engineers were blamed, I was blamed, the customer was adamant that they had never, would never, could never have done anything that could have caused it. When we solved the problem and proved it was their code I don't recall anybody even saying thanks.
So all in all I'm not sure I'd recommend doing a regular raw snapshot in the background.
tim
On Oct 7, 2005, at 12:08 PM, Andreas Raab wrote:
You can *not* get away without a GC, simply because the GC state isn't preserved across snapshots. We actually had some obscure crashes related to this problem in the past (anyone remember the crashes caused by the root bit being set on the active context after a snapshot?)
In other words, the full GC is required to save the image in a well- known "right-after-full-gc-state". One would (at least) have to save sufficient information for the garbage collector to make this work. Likely other fixes too, since a full garbage collect cleans up many, many things that we usually don't even think about (cached contexts for example).
Ok, thanks, good to know.
Now, it should still work to do what Tom was suggesting, right? Do the fullGC in the parent process, then immediately fork and snapshot (without doing a second GC in the child). Does the order of, eg, #storeContextRegisters: and #fullGC inside #snapshot: matter?
Avi
Mmmm, I have thought about doing a full GC without compacting, that would save time. Anyone consider if that would work?
On 7-Oct-05, at 12:49 PM, Avi Bryant wrote:
On Oct 7, 2005, at 12:08 PM, Andreas Raab wrote:
You can *not* get away without a GC, simply because the GC state isn't preserved across snapshots. We actually had some obscure crashes related to this problem in the past (anyone remember the crashes caused by the root bit being set on the active context after a snapshot?)
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On 7-Oct-05, at 12:59 PM, John M McIntosh wrote:
Mmmm, I have thought about doing a full GC without compacting, that would save time. Anyone consider if that would work?
Well, to do a full gc we have to scan the entire object memory, mark&sweep style, and compacting after that is what gets the dead memory out of the way. Leaving free chunks lying around might not actually kill you but it seems a bit pointless. You're going to have to do it at some point anyway.
tim
On Oct 7, 2005, at 1:24 PM, tim Rowledge wrote:
On 7-Oct-05, at 12:59 PM, John M McIntosh wrote:
Mmmm, I have thought about doing a full GC without compacting, that would save time. Anyone consider if that would work?
Well, to do a full gc we have to scan the entire object memory, mark&sweep style, and compacting after that is what gets the dead memory out of the way. Leaving free chunks lying around might not actually kill you but it seems a bit pointless. You're going to have to do it at some point anyway.
Well, for my very specific goal of writing an image in the background it would be very useful... I'm happy to do the compacting at some point anyway, as long as it's not during the few seconds that the OS is trying its best to run two processes concurrently using the same memory pages.
Avi
Hello,
Mmmm, I have thought about doing a full GC without compacting, that would save time. Anyone consider if that would work?
Well, to do a full gc we have to scan the entire object memory, mark&sweep style, and compacting after that is what gets the dead memory out of the way. Leaving free chunks lying around might not actually kill you but it seems a bit pointless. You're going to have to do it at some point anyway.
Well, for my very specific goal of writing an image in the background it would be very useful... I'm happy to do the compacting at some point anyway, as long as it's not during the few seconds that the OS is trying its best to run two processes concurrently using the same memory pages.
Mark&sweep phase mutates the object headers, so almost of virtual memory pages (4k bytes or so) are "modified" even if you avoid the compaction. Not doing compaction wouldn't change it.
Having a "full GC, fork and let the child write image to file" should work, and the set of pages tainted in the period of concurrent run may not be so big, actually.
-- Yoshiki
On Oct 7, 2005, at 2:41 PM, Yoshiki Ohshima wrote:
Mark&sweep phase mutates the object headers, so almost of virtual memory pages (4k bytes or so) are "modified" even if you avoid the compaction. Not doing compaction wouldn't change it.
Having a "full GC, fork and let the child write image to file" should work, and the set of pages tainted in the period of concurrent run may not be so big, actually.
Yeah, it doesn't seem to be. I haven't done any detailed analysis, but just watching the output of top, if you take out the GC basically all you see is a chunk of memory being moved from the "resident" pile to the "shared" pile and then back again when the child process dies. Whereas with the GC in the child, you get a large increase in total memory use for that period.
Avi
Avi,
Mark&sweep phase mutates the object headers, so almost of virtual memory pages (4k bytes or so) are "modified" even if you avoid the compaction. Not doing compaction wouldn't change it.
Having a "full GC, fork and let the child write image to file" should work, and the set of pages tainted in the period of concurrent run may not be so big, actually.
Yeah, it doesn't seem to be. I haven't done any detailed analysis, but just watching the output of top, if you take out the GC basically all you see is a chunk of memory being moved from the "resident" pile to the "shared" pile and then back again when the child process dies. Whereas with the GC in the child, you get a large increase in total memory use for that period.
I may have misunderstood the problem, but I might still imagine that adding a primitive that does
"full GC, fork and let the child write image to file"
"at once" (not returning the control to the image level on both child and parent) may give some benefit.
-- Yoshiki
On Oct 7, 2005, at 3:40 PM, Yoshiki Ohshima wrote:
I may have misunderstood the problem, but I might still imagine that adding a primitive that does
"full GC, fork and let the child write image to file"
"at once" (not returning the control to the image level on both child and parent) may give some benefit.
Yes, that's exactly what I'm trying. It seems to work ok.
Avi
Of course if you have a spare GB or two, you could do the fullGC ask for a block of memory, copy start of memory to end of memory there, then write that out at your leisure, that forgoes the need for fork and the like. Assumes of course you can ask and get 500MB at no cost. Surely the write happens quite quickly on modern high-speed disk i/o machines tho?
Seems to me if you're wanting to minimize downtime you fork and do a snapshot, that as noticed takes upwards of image size to do. Otherwise as you say full GC, fork for the write, but does that save anything?
On 7-Oct-05, at 3:40 PM, Yoshiki Ohshima wrote:
I may have misunderstood the problem, but I might still imagine that adding a primitive that does
"full GC, fork and let the child write image to file"
"at once" (not returning the control to the image level on both child and parent) may give some benefit.
-- Yoshiki
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On Oct 7, 2005, at 4:09 PM, John M McIntosh wrote:
Of course if you have a spare GB or two, you could do the fullGC ask for a block of memory, copy start of memory to end of memory there, then write that out at your leisure, that forgoes the need for fork and the like. Assumes of course you can ask and get 500MB at no cost. Surely the write happens quite quickly on modern high-speed disk i/ o machines tho? Seems to me if you're wanting to minimize downtime you fork and do a snapshot, that as noticed takes upwards of image size to do. Otherwise as you say full GC, fork for the write, but does that save anything?
The scenario is that you have a server packed with large images that you want saving to disk periodically. So mostly what I'm trying to optimize is memory usage, but without unreasonable delay by the standards of web applications. The two options that take more or less constant memory seem to be a regular snapshot, or GC->fork-
snapshot. For a largish image (100-200MB), a GC takes maybe 2s
(acceptable, if annoying) whereas a snapshot takes maybe 15s (pretty much unacceptable). So the forking seems like the right choice. Though if we could get rid of the GC altogether that would be great...
Avi
On Sat, 08 Oct 2005 01:34:04 +0200, Avi Bryant avi.bryant@gmail.com wrote:
On Oct 7, 2005, at 4:09 PM, John M McIntosh wrote:
Of course if you have a spare GB or two, you could do the fullGC ask for a block of memory, copy start of memory to end of memory there, then write that out at your leisure, that forgoes the need for fork and the like. Assumes of course you can ask and get 500MB at no cost. Surely the write happens quite quickly on modern high-speed disk i/o machines tho? Seems to me if you're wanting to minimize downtime you fork and do a snapshot, that as noticed takes upwards of image size to do. Otherwise as you say full GC, fork for the write, but does that save anything?
The scenario is that you have a server packed with large images that you want saving to disk periodically. So mostly what I'm trying to optimize is memory usage, but without unreasonable delay by the standards of web applications. The two options that take more or less constant memory seem to be a regular snapshot, or GC->fork- >snapshot. For a largish image (100-200MB), a GC takes maybe 2s (acceptable, if annoying) whereas a snapshot takes maybe 15s (pretty much unacceptable). So the forking seems like the right choice. Though if we could get rid of the GC altogether that would be great...
I believe that one of the reasons for GC standing in the way of such an application is the lack of integrating it with the virtual memory paging mechanism. And 100-200BM is not really that much, IMHO. So it might be time for considering an implementation of Squeak based on
http://os.inf.tu-dresden.de/fiasco/
/Klaus
Avi
Avi, so does the write to disk actually take 12 seconds? You could look at writeImageFileIO or up in writeImageFile and stick a fork there and have the child do the I/O. A bit further away from the OO space at this point so it's clearer what you are doing perhaps.
On 7-Oct-05, at 4:34 PM, Avi Bryant wrote:
On Oct 7, 2005, at 4:09 PM, John M McIntosh wrote:
Of course if you have a spare GB or two, you could do the fullGC ask for a block of memory, copy start of memory to end of memory there, then write that out at your leisure, that forgoes the need for fork and the like. Assumes of course you can ask and get 500MB at no cost. Surely the write happens quite quickly on modern high-speed disk i/ o machines tho? Seems to me if you're wanting to minimize downtime you fork and do a snapshot, that as noticed takes upwards of image size to do. Otherwise as you say full GC, fork for the write, but does that save anything?
The scenario is that you have a server packed with large images that you want saving to disk periodically. So mostly what I'm trying to optimize is memory usage, but without unreasonable delay by the standards of web applications. The two options that take more or less constant memory seem to be a regular snapshot, or GC-
fork->snapshot. For a largish image (100-200MB), a GC takes maybe
2s (acceptable, if annoying) whereas a snapshot takes maybe 15s (pretty much unacceptable). So the forking seems like the right choice. Though if we could get rid of the GC altogether that would be great...
Avi
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
Avi -
Try this: Instead of "self fullGC" in primSnapshot, try inserting:
"DISCLAIMER: This requires some careful review which I don't have the time for right now. This is NOT working or tested code so don't even think of submitting this to VMMaker unless you actually understand why this ought to work and what differences there possibly might be between a full GC and an incremental GC + tenuring."
savedThreshold := tenuringThreshold. "remember prior threshold" tenuringThreshold := 0. "reset tenuring threshold" self incrementalGC. "force tenuring" tenuringThreshold := savedThreshold.
This may already be enough since tenuring will clean up the roots and thus leave the image in a stable state. But of course, it does require the image to perform a full GC if you want to reclaim space before the snapshot.
Cheers, - Andreas
Avi Bryant wrote:
On Oct 7, 2005, at 4:09 PM, John M McIntosh wrote:
Of course if you have a spare GB or two, you could do the fullGC ask for a block of memory, copy start of memory to end of memory there, then write that out at your leisure, that forgoes the need for fork and the like. Assumes of course you can ask and get 500MB at no cost. Surely the write happens quite quickly on modern high-speed disk i/ o machines tho? Seems to me if you're wanting to minimize downtime you fork and do a snapshot, that as noticed takes upwards of image size to do. Otherwise as you say full GC, fork for the write, but does that save anything?
The scenario is that you have a server packed with large images that you want saving to disk periodically. So mostly what I'm trying to optimize is memory usage, but without unreasonable delay by the standards of web applications. The two options that take more or less constant memory seem to be a regular snapshot, or GC->fork- >snapshot. For a largish image (100-200MB), a GC takes maybe 2s (acceptable, if annoying) whereas a snapshot takes maybe 15s (pretty much unacceptable). So the forking seems like the right choice. Though if we could get rid of the GC altogether that would be great...
Avi
Actually a bit cleaner usage would be to use the forceTenureFlag
set to true before doing the incrementalGC. it's looked for and reset in the GC logic.
Also see primitiveForceTenure
On 7-Oct-05, at 5:37 PM, Andreas Raab wrote:
Avi -
Try this: Instead of "self fullGC" in primSnapshot, try inserting:
"DISCLAIMER: This requires some careful review which I don't have the time for right now. This is NOT working or tested code so don't even think of submitting this to VMMaker unless you actually understand why this ought to work and what differences there possibly might be between a full GC and an incremental GC + tenuring." savedThreshold := tenuringThreshold. "remember prior threshold" tenuringThreshold := 0. "reset tenuring threshold" self incrementalGC. "force tenuring" tenuringThreshold := savedThreshold.This may already be enough since tenuring will clean up the roots and thus leave the image in a stable state. But of course, it does require the image to perform a full GC if you want to reclaim space before the snapshot.
Cheers,
- Andreas
Avi Bryant wrote:
On Oct 7, 2005, at 4:09 PM, John M McIntosh wrote:
Of course if you have a spare GB or two, you could do the fullGC ask for a block of memory, copy start of memory to end of memory there, then write that out at your leisure, that forgoes the need for fork and the like. Assumes of course you can ask and get 500MB at no cost. Surely the write happens quite quickly on modern high-speed disk i/ o machines tho? Seems to me if you're wanting to minimize downtime you fork and do a snapshot, that as noticed takes upwards of image size to do. Otherwise as you say full GC, fork for the write, but does that save anything?
The scenario is that you have a server packed with large images that you want saving to disk periodically. So mostly what I'm trying to optimize is memory usage, but without unreasonable delay by the standards of web applications. The two options that take more or less constant memory seem to be a regular snapshot, or GC->fork- >snapshot. For a largish image (100-200MB), a GC takes maybe 2s (acceptable, if annoying) whereas a snapshot takes maybe 15s (pretty much unacceptable). So the forking seems like the right choice. Though if we could get rid of the GC altogether that would be great... Avi
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On Oct 7, 2005, at 5:37 PM, Andreas Raab wrote:
Avi -
Try this: Instead of "self fullGC" in primSnapshot, try inserting:
"DISCLAIMER: This requires some careful review which I don't have the time for right now. This is NOT working or tested code so don't even think of submitting this to VMMaker unless you actually understand why this ought to work and what differences there possibly might be between a full GC and an incremental GC + tenuring." savedThreshold := tenuringThreshold. "remember prior threshold" tenuringThreshold := 0. "reset tenuring threshold" self incrementalGC. "force tenuring" tenuringThreshold := savedThreshold.This may already be enough since tenuring will clean up the roots and thus leave the image in a stable state. But of course, it does require the image to perform a full GC if you want to reclaim space before the snapshot.
Excellent, thanks. Do you have any tests to suggest? Any edge cases to look at? Assuming it seems to basically work, how could I assure myself that this was an ok thing to put into production?
Avi
On Oct 7, 2005, at 5:37 PM, Andreas Raab wrote:
Avi -
Try this: Instead of "self fullGC" in primSnapshot, try inserting:
"DISCLAIMER: This requires some careful review which I don't have the time for right now. This is NOT working or tested code so don't even think of submitting this to VMMaker unless you actually understand why this ought to work and what differences there possibly might be between a full GC and an incremental GC + tenuring." savedThreshold := tenuringThreshold. "remember prior threshold" tenuringThreshold := 0. "reset tenuring threshold" self incrementalGC. "force tenuring" tenuringThreshold := savedThreshold.This may already be enough since tenuring will clean up the roots and thus leave the image in a stable state. But of course, it does require the image to perform a full GC if you want to reclaim space before the snapshot.
For what it's worth (probably not much) I've been running for the last several days with this in place, along with a fork before writing out the image, and haven't seen any ill effects. Here's the full #snapshot: I'm using:
snapshot: embedded "update state of active context" | activeProc dataSize rcvr setMacType savedThreshold | compilerInitialized ifTrue: [self compilerPreSnapshot] ifFalse: [self storeContextRegisters: activeContext].
"update state of active process" activeProc _ self fetchPointer: ActiveProcessIndex ofObject: self schedulerPointer. self storePointer: SuspendedContextIndex ofObject: activeProc withValue: activeContext.
"compact memory and compute the size of the memory actually in use" self incrementalGC.
"DISCLAIMER: This requires some careful review which I don't have the time for right now. This is NOT working or tested code so don't even think of submitting this to VMMaker unless you actually understand why this ought to work and what differences there possibly might be between a full GC and an incremental GC + tenuring."
savedThreshold _ tenuringThreshold. "remember prior threshold" tenuringThreshold _ 0. "reset tenuring threshold" self incrementalGC. "force tenuring" tenuringThreshold _ savedThreshold.
self snapshotCleanUp.
dataSize _ freeBlock - self startOfMemory. "Assume all objects are below the start of the free block" successFlag ifTrue: [rcvr _ self popStack. "pop rcvr" self push: trueObj. (self cCode: 'fork()') = 0 ifTrue: [self writeImageFile: dataSize. self cCode: '_exit(0)'].
embedded ifFalse: ["set Mac file type and creator; this is a noop on other platforms" setMacType _ self ioLoadFunction: 'setMacFileTypeAndCreator' From: 'FilePlugin'. setMacType = 0 ifFalse: [self cCode: '((int (*) (char*, char*, char*)) setMacType) (getImageName(), "STim", "FAST")']]. self pop: 1].
"activeContext was unmarked in #snapshotCleanUp, mark it old " self beRootIfOld: activeContext. successFlag ifTrue: [self push: falseObj] ifFalse: [self push: rcvr]. compilerInitialized ifTrue: [self compilerPostSnapshot]
Andreas wrote...
Try this: Instead of "self fullGC" in primSnapshot, try inserting:
"DISCLAIMER: This requires some careful review which I don't have the time for right now. This is NOT working or tested code so don't even think of submitting this to VMMaker unless you actually understand why this ought to work and what differences there possibly might be between a full GC and an incremental GC + tenuring."
savedThreshold := tenuringThreshold. "remember prior threshold" tenuringThreshold := 0. "reset tenuring threshold" self incrementalGC. "force tenuring" tenuringThreshold := savedThreshold.
I tuned into this discussion a bit late, but this is close to what I was going to recommend.
Here are some thoughts...
1. Given a stable application running for hours, I would *not* expect compaction to require a lot of movement after one or two full GCs. The old objects should just pile up at the bottom of memory, and the compaction phase shouldn't take much longer than that of an incremental GC. If this is not happening, then I'd say it's worth a little research.
2. I thought we had code around to reconstruct the roots table from the root bits. If so, it should be possible to just do an incremental GC and then save the image, *provided* that that image is somehow flagged as requiring root table reconstruction (this sounds like something my dentist does) before resuming.
3. The one argument against tenuring is that it causes creep toward a full GC. So if you keep tenuring, you will have to do a full GC from time to time. Whereas, I'm assuming that a stable server application will run essentially for ever without a full GC being necessary (if someone has added gratuitous tenuring somewhere, you may want to turn it off). If you can avoid tenuring, you should be able to avoid the worst case latency.
4. If you have access to the copy-on write logic, there ought to be a way to write only the altered pages in subsequent saves. This would make for a much quicker checkpoint, but it will also leave you with a messy save format to decode when you want to restart from that checkpoint. It's the kind of thing the OS itself ought to do for you. Really ;-)
Ciao - Dan
On Fri, Oct 07, 2005 at 04:34:04PM -0700, Avi Bryant wrote:
The scenario is that you have a server packed with large images that you want saving to disk periodically. So mostly what I'm trying to optimize is memory usage, but without unreasonable delay by the standards of web applications. The two options that take more or less constant memory seem to be a regular snapshot, or GC->fork-
snapshot. For a largish image (100-200MB), a GC takes maybe 2s
(acceptable, if annoying) whereas a snapshot takes maybe 15s (pretty much unacceptable). So the forking seems like the right choice. Though if we could get rid of the GC altogether that would be great...
Hi Avi,
With the configuration you are describing, it might help a little bit if you do the background image saves at lower OS scheduling priority. This should reduce the impact of the image save processes on responsiveness of the Seaside servers.
The attached change set does this. You'll need to dust off your VMMaker to use it.
From the preamble:
Adds #primitiveNice to OSPP to allow changing the scheduling priority of the current OS process (the Squeak VM process).
Updates #saveImageInBackground to perform the background image save at lower scheduling priority.
Apply this change set to OSProcess 4.0 and OSProcessPlugin 4.0 (i.e. the latest SqueakMap versions). XDisplayControlPlugin 2.0 and AioPlugin 2.0 should also be included in the VMMaker build (AioPlugin just so all the OSP unit tests will pass).
HTH,
Dave
p.s. Sorry about the .cs format. I'm using Monticello now (great stuff!) but I'm still learning how to use it properly, e.g. to make diffs. Next time I'll send an .mcz.
Yes but the ugly bit from memory cache, bandwidth etc point is having to move all the bytes around. I was thinking if only did that every other time or something it might gain you a bit of performance.
However on another note what I've observed with squeak at idle is a lot of time taken to do newspace GC work. Seems we allocate a lot of contexts, which then count towards the 4000 object allocation trigger and we force a GC, which disposes of the contexts on the free chain, which forces us to reallocate them. I'm not sure why we are allocating new contexts all the time, versus finding one on the free context tree, which seems mostly empty.
I did cheat and say, if I allocate one for the context, then decrement the object allocation counter, but that didn't alter the observed behaviour much.
Mmm perhaps I need to poke at it some more to understand what is happening.
On 7-Oct-05, at 1:24 PM, tim Rowledge wrote:
On 7-Oct-05, at 12:59 PM, John M McIntosh wrote:
Mmmm, I have thought about doing a full GC without compacting, that would save time. Anyone consider if that would work?
Well, to do a full gc we have to scan the entire object memory, mark&sweep style, and compacting after that is what gets the dead memory out of the way. Leaving free chunks lying around might not actually kill you but it seems a bit pointless. You're going to have to do it at some point anyway.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On 7-Oct-05, at 12:49 PM, Avi Bryant wrote:
On Oct 7, 2005, at 12:08 PM, Andreas Raab wrote:
You can *not* get away without a GC, simply because the GC state isn't preserved across snapshots. We actually had some obscure crashes related to this problem in the past (anyone remember the crashes caused by the root bit being set on the active context after a snapshot?)
In other words, the full GC is required to save the image in a well-known "right-after-full-gc-state". One would (at least) have to save sufficient information for the garbage collector to make this work. Likely other fixes too, since a full garbage collect cleans up many, many things that we usually don't even think about (cached contexts for example).
Ok, thanks, good to know.
Now, it should still work to do what Tom was suggesting, right? Do the fullGC in the parent process, then immediately fork and snapshot (without doing a second GC in the child). Does the order of, eg, #storeContextRegisters: and #fullGC inside #snapshot: matter?
Avi
If you do the fullGC, then the fork and snapshot which also does a fullGC, then cleans up context records and information about primitive linkups you should find that the amount of memory altered is much less. The problem with doing a snapshot at any point is that it most likely will move *most* memory in the image as it compacts the image as a result of the full GC. Back to Back full GCs won't move anything (depending on if the interpreter runs between calls). Thus altering much less memory.
However isn't there a cost here in the parent process of doing the fullGC, those take time.
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
John M McIntosh writes:
If you do the fullGC, then the fork and snapshot which also does a fullGC, then cleans up context records and information about primitive linkups you should find that the amount of memory altered is much less. The problem with doing a snapshot at any point is that it most likely will move *most* memory in the image as it compacts the image as a result of the full GC. Back to Back full GCs won't move anything (depending on if the interpreter runs between calls). Thus altering much less memory.
However the fullGC would still tag every live object thus forcing the pages to be copied.
Couldn't the fullGC be replaced by a scan clearing root objects? That would only need to change the roots not everything. There'll never be that many root objects.
If the global GC that removes the root method context's is done in the image what stops the code between it and the snapshot from promoting another method context? I can see how an image side GC could reduce the size of the risk, but unless no GCs happen between the full GC and the snapshot surely there's still some risk now?
Bryce
On Oct 7, 2005, at 1:32 PM, John M McIntosh wrote:
If you do the fullGC, then the fork and snapshot which also does a fullGC, then cleans up context records and information about primitive linkups you should find that the amount of memory altered is much less. The problem with doing a snapshot at any point is that it most likely will move *most* memory in the image as it compacts the image as a result of the full GC. Back to Back full GCs won't move anything (depending on if the interpreter runs between calls). Thus altering much less memory.
Ok. That's definitely the safest option...
However isn't there a cost here in the parent process of doing the fullGC, those take time.
Yep. But relatively little time compared to a full snapshot, so it's still a win.
Avi
John M McIntosh wrote:
On 7-Oct-05, at 12:49 PM, Avi Bryant wrote:
Now, it should still work to do what Tom was suggesting, right? Do the fullGC in the parent process, then immediately fork and snapshot (without doing a second GC in the child). Does the order of, eg, #storeContextRegisters: and #fullGC inside #snapshot: matter?
If you do the fullGC, then the fork and snapshot which also does a fullGC, then cleans up context records and information about primitive linkups you should find that the amount of memory altered is much less.
It's not. Garbage collection touches (modifies, writes) the header bits of the objects touched which causes the pages to be marked dirty and allocated. You'd have to avoid the entire GC to make this work.
Cheers, - Andreas
Duh, need coffee forget all about mark bit.... Which of course faults the entire Virtual memory page. Still not having to actually shuffle all the memory a few bytes closer to 0000000 should save a bit of time.
On 7-Oct-05, at 2:40 PM, Andreas Raab wrote:
John M McIntosh wrote:
On 7-Oct-05, at 12:49 PM, Avi Bryant wrote:
Now, it should still work to do what Tom was suggesting, right? Do the fullGC in the parent process, then immediately fork and snapshot (without doing a second GC in the child). Does the order of, eg, #storeContextRegisters: and #fullGC inside #snapshot: matter?
If you do the fullGC, then the fork and snapshot which also does a fullGC, then cleans up context records and information about primitive linkups you should find that the amount of memory altered is much less.
It's not. Garbage collection touches (modifies, writes) the header bits of the objects touched which causes the pages to be marked dirty and allocated. You'd have to avoid the entire GC to make this work.
Cheers,
- Andreas
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
Hello all,
Does anyone have any suggestions for relationship mapping documentation? I've been keeping track of a bunch of information like people, organizations, grants, funding projects, legislation, web links ... all on Visio.
Does anyone know of anything that can make this easier? Is there some system out there that will help me track and query for relationships, and display it all in a relationship map?
Any suggestions would be welcome even if it's just close.
Ron
Ron Teitelbaum wrote:
Hello all,
Does anyone have any suggestions for relationship mapping documentation? I've been keeping track of a bunch of information like people, organizations, grants, funding projects, legislation, web links ... all on Visio.
Does anyone know of anything that can make this easier? Is there some system out there that will help me track and query for relationships, and display it all in a relationship map?
Any suggestions would be welcome even if it's just close.
Ron
I always wanted to implement something like this: http://www.thebrain.com/lowbrow/default.htm in squeak. Ok.. there's a patent... so maybe something _like_ Personal Brain.
They could have done a lot with it, but chose not to pursue the personal version. I think they missed a big opportunity.
Brad Fuller wrote:
Ron Teitelbaum wrote:
Hello all,
Does anyone have any suggestions for relationship mapping documentation? I've been keeping track of a bunch of information like people, organizations, grants, funding projects, legislation, web links ... all on Visio.
Does anyone know of anything that can make this easier? Is there some system out there that will help me track and query for relationships, and display it all in a relationship map?
Any suggestions would be welcome even if it's just close.
Ron
I always wanted to implement something like this: http://www.thebrain.com/lowbrow/default.htm in squeak. Ok.. there's a patent... so maybe something _like_ Personal Brain.
They could have done a lot with it, but chose not to pursue the personal version. I think they missed a big opportunity.
oh... see: http://www.thebrain.com/ for the demo
-----Original Message----- From: squeak-dev-bounces@lists.squeakfoundation.org [mailto:squeak-dev-bounces@lists.squeakfoundation.org] On Behalf Of Brad Fuller Sent: Friday, October 07, 2005 6:31 PM To: The general-purpose Squeak developers list Subject: Re: Knowledge Documentation topic/relationship maps??
(snip)
oh... see: http://www.thebrain.com/ for the demo
Speaking of relationship maps... Are there any Squeak implementations of this thing performed by the Java app at www.thebrain.com
Other example here:
I'm not sure what these are properly called...but I want to experiment with them in the Squeak environment.
-- kurt
Wow!!!!
I am really impressed. I used the google touch map put in a url and it built a map automatically almost exactly like the one I've been building!!!. What a great tool this is. Thanks for pointing it out!
Thank you Kurt!!!
-----Original Message----- From: squeak-dev-bounces@lists.squeakfoundation.org [mailto:squeak-dev-bounces@lists.squeakfoundation.org] On Behalf Of Kurt Thams Sent: Monday, October 10, 2005 7:09 PM To: 'The general-purpose Squeak developers list' Subject: topic/relationship maps in Squeak?
-----Original Message----- From: squeak-dev-bounces@lists.squeakfoundation.org [mailto:squeak-dev-bounces@lists.squeakfoundation.org] On Behalf Of Brad Fuller Sent: Friday, October 07, 2005 6:31 PM To: The general-purpose Squeak developers list Subject: Re: Knowledge Documentation topic/relationship maps??
(snip)
oh... see: http://www.thebrain.com/ for the demo
Speaking of relationship maps... Are there any Squeak implementations of this thing performed by the Java app at www.thebrain.com
Other example here:
I'm not sure what these are properly called...but I want to experiment with them in the Squeak environment.
-- kurt
Is this along the lines of what you're looking for?
http://www.conceptdraw.com/en/products/mindmap/main.php
- Daniel
On Oct 7, 2005, at 9:16 PM, Brad Fuller wrote:
Ron Teitelbaum wrote:
Hello all,
Does anyone have any suggestions for relationship mapping documentation? I've been keeping track of a bunch of information like people, organizations, grants, funding projects, legislation, web links ... all on Visio.
Does anyone know of anything that can make this easier? Is there some system out there that will help me track and query for relationships, and display it all in a relationship map?
Any suggestions would be welcome even if it's just close.
Ron
I always wanted to implement something like this: http://www.thebrain.com/lowbrow/default.htm in squeak. Ok.. there's a patent... so maybe something _like_ Personal Brain.
They could have done a lot with it, but chose not to pursue the personal version. I think they missed a big opportunity.
-- Brad Fuller (408) 799-6124 ** Sonaural Audio Studios ** (408) 799-6123 West San Jose (408) 799-6124 Cambrian ________________________________ Hear us online: www.Sonaural.com See me on O'Reilly: http://www.oreillynet.com/pub/au/2184
Daniel Salama wrote:
Is this along the lines of what you're looking for?
I didn't know they trademarked "Mind map" and "mind mapping"!
There's the free version as well: http://freemind.sourceforge.net/wiki/index.php/Main_Page
Brad,
Hey this one is really pretty cool!
Thank you!!!
Ron
-----Original Message----- From: squeak-dev-bounces@lists.squeakfoundation.org [mailto:squeak-dev-bounces@lists.squeakfoundation.org] On Behalf Of Brad Fuller Sent: Saturday, October 08, 2005 12:48 PM To: The general-purpose Squeak developers list Subject: Re: Knowledge Documentation topic/relationship maps??
Daniel Salama wrote:
Is this along the lines of what you're looking for?
I didn't know they trademarked "Mind map" and "mind mapping"!
There's the free version as well: http://freemind.sourceforge.net/wiki/index.php/Main_Page
squeak-dev@lists.squeakfoundation.org
-
Andreas Raab -
Avi Bryant -
Avi Bryant -
Brad Fuller -
Bryce Kampjes -
Craig Latta -
Dan Ingalls -
Daniel Salama -
David T. Lewis -
John M McIntosh -
Klaus D. Witzel -
Kurt Thams -
Ron Teitelbaum -
tim Rowledge -
Tom Phoenix -
Yoshiki Ohshima