Unfortunately there is no easy fix , which will fit all possible use scenarios. As you can see from comments in code, this feature was disabled on purpose for cases when your application works extensively with external resources/libraries, which using malloc() a lot.
You mean FFI? Someone took the trouble to write this for the manpage. My gut thinks FFI has been around longer than the releasing of memory feature -- not sure.
-mmap size[mk] requests that a variable heap of at most size bytes be allo‐ cated. (The suffixes are as described for the '-memory' option.) squeak will initially allocate a heap that is large enough to hold the image, with a small amount of headroom. If at any time Squeak requires more memory for its image then addi‐ tional space will be allocated dynamically. Likewise, when mem‐ ory is no longer needed it will deallocated and returned to the system. The size argument places an upper limit on how big the heap can grow in this fashion. squeak uses a dynamic heap by default with the maximum size set to 75% of the available vir‐ tual memory or 1 gigabyte, whichever is smaller.
Eliot, is the useMmap variable you referred to set by this argument? I tried launching my image with -mmap 700M but it still didn't seem to want to deallocate.
Same with the Classic VM.
On 22 October 2012 04:35, Chris Muller asqueaker@gmail.com wrote:
Unfortunately there is no easy fix , which will fit all possible use scenarios. As you can see from comments in code, this feature was disabled on purpose for cases when your application works extensively with external resources/libraries, which using malloc() a lot.
You mean FFI? Someone took the trouble to write this for the manpage. My gut thinks FFI has been around longer than the releasing of memory feature -- not sure.
-mmap size[mk] requests that a variable heap of at most size bytes be allo‐ cated. (The suffixes are as described for the '-memory' option.) squeak will initially allocate a heap that is large enough to hold the image, with a small amount of headroom. If at any time Squeak requires more memory for its image then addi‐ tional space will be allocated dynamically. Likewise, when mem‐ ory is no longer needed it will deallocated and returned to the system. The size argument places an upper limit on how big the heap can grow in this fashion. squeak uses a dynamic heap by default with the maximum size set to 75% of the available vir‐ tual memory or 1 gigabyte, whichever is smaller.Eliot, is the useMmap variable you referred to set by this argument? I tried launching my image with -mmap 700M but it still didn't seem to want to deallocate.
Same with the Classic VM.
It is controlled by overallocateMemory flag, which is not exposed via command-line options.
Here the patched file. I added a new option -releaseUnusedMemory to turn this flag on.
I didn't tried to build VM with it, so don't blame me if it doesn't works :)
diff --git a/platforms/unix/vm/sqUnixMain.c b/platforms/unix/vm/sqUnixMain.c index a593b2a..875bf0d 100644 --- a/platforms/unix/vm/sqUnixMain.c +++ b/platforms/unix/vm/sqUnixMain.c @@ -98,6 +98,7 @@ static char **squeakArgVec= 0;
static int extraMemory= 0; int useMmap= DefaultMmapSize * 1024 * 1024; +extern int overallocateMemory; /* defined in vm/sqUnixMemory.c */
static int useItimer= 1; /* 0 to disable itimer-based clock */ static int installHandlers= 1; /* 0 to disable sigusr1 & sigsegv handlers */ @@ -1263,6 +1264,7 @@ static int vm_parseArgument(int argc, char **argv) /* vm arguments */
if (!strcmp(argv[0], "-help")) { usage(); return 1; } + else if (!strcmp(argv[0], "-releaseUnusedMemory") { overallocateMemory = 1; return 1; } else if (!strcmp(argv[0], "-noevents")) { noEvents = 1; return 1; } else if (!strcmp(argv[0], "-nomixer")) { noSoundMixer = 1; return 1; } else if (!strcmp(argv[0], "-notimer")) { useItimer = 0; return 1; }
On Sun, Oct 21, 2012 at 09:35:15PM -0500, Chris Muller wrote:
Unfortunately there is no easy fix , which will fit all possible use scenarios. As you can see from comments in code, this feature was disabled on purpose for cases when your application works extensively with external resources/libraries, which using malloc() a lot.
You mean FFI? Someone took the trouble to write this for the manpage. My gut thinks FFI has been around longer than the releasing of memory feature -- not sure.
-mmap size[mk] requests that a variable heap of at most size bytes be allo??? cated. (The suffixes are as described for the '-memory' option.) squeak will initially allocate a heap that is large enough to hold the image, with a small amount of headroom. If at any time Squeak requires more memory for its image then addi??? tional space will be allocated dynamically. Likewise, when mem??? ory is no longer needed it will deallocated and returned to the system. The size argument places an upper limit on how big the heap can grow in this fashion. squeak uses a dynamic heap by default with the maximum size set to 75% of the available vir??? tual memory or 1 gigabyte, whichever is smaller.Eliot, is the useMmap variable you referred to set by this argument? I tried launching my image with -mmap 700M but it still didn't seem to want to deallocate.
Same with the Classic VM.
Eliot is referring to Ian Piumarta's explanation in sqUnixMemory.c here:
/* Note: * * The code allows memory to be overallocated; i.e., the initial * block is reserved via mmap() and then the unused portion * munmap()ped from the top end. This is INHERENTLY DANGEROUS since * malloc() may randomly map new memory in the block we "reserved" * and subsequently unmap()ped. Enabling this causes crashes in * Croquet, which makes heavy use of the FFI and thus calls malloc() * all over the place. * * For this reason, overallocateMemory is DISABLED by default. * * The upshot of all this is that Squeak will claim (and hold on to) * ALL of the available virtual memory (or at least 75% of it) when * it starts up. If you can't live with that, use the -memory * option to allocate a fixed size heap. */
See also the man page for sbrk for some related cautions about mixing malloc with lower level allocation/deallocation calls.
Igor just posted a patch to let you override the default for the command line, but you should be very sure that your application is not doing malloc calls. Plugins (e.g. OSProcess) and FFI calls are entirely free to do this, and you should assume that they will do so unless you positively know that they do not.
Chris, may I ask why you see this as a problem? Is there some practical situation in which you are running out of real system memory as a consequence of the memory mapping strategy?
<OT> I'll note for the record that Java VMs seem to adopt a much simpler strategy for object memory management: Whenever you run low on memory, they crash. Yet somehow people manage to build commercial systems on top of this. Go figure. </OT>
Dave
Eliot is referring to Ian Piumarta's explanation in sqUnixMemory.c here:
/* Note: * * The code allows memory to be overallocated; i.e., the initial * block is reserved via mmap() and then the unused portion * munmap()ped from the top end. This is INHERENTLY DANGEROUS since * malloc() may randomly map new memory in the block we "reserved" * and subsequently unmap()ped. Enabling this causes crashes in * Croquet, which makes heavy use of the FFI and thus calls malloc() * all over the place. * * For this reason, overallocateMemory is DISABLED by default. * * The upshot of all this is that Squeak will claim (and hold on to) * ALL of the available virtual memory (or at least 75% of it) when * it starts up. If you can't live with that, use the -memory * option to allocate a fixed size heap. */See also the man page for sbrk for some related cautions about mixing malloc with lower level allocation/deallocation calls.
If the VM starts wth a block initially allocated for the object-memory, and extended as needed for subsequent allocation (String new: 4000000), but THEN reduced once not needed -- why would a *subsequent* malloc call grabbing memory from the part that was deallocated cause a crash? At first I thought the crash would occur after a subsequent extension of object-memory into memory that was malloc'd for another purpose (plugin) but.. that could happen on the _first_ time it is extended, couldn't it?
IOW:
I start out using 40MB for object memory.
0----------------------40MB
Plugins make some malloc calls, which allocate somewhere just above..
0----------------------40MB-----+5MB(malloc'd)
Now I need more allocation for object memory, so I'm trampling what the plugin malloc'd?
Sorry, I unfortunately do not know C at all, thanks for helping me understand the situation.
Chris, may I ask why you see this as a problem? Is there some practical situation in which you are running out of real system memory as a consequence of the memory mapping strategy?
I have a Squeak-based server where 99% of requests are small, but once a big request comes in, that server image jumps from 40MB to 400MB. The extra memory was only needed for 10 seconds but unfortunately sits there unused forever-after. So it's almost like a memory-leak where the only way to recover it is to bounce the server.
What would be worse is if the over-allocated image then affects GC performance. I'm still trying to figure out whether this is the case -- it seems like not thank goodness. Do you know for sure the answer to that? IOW should full GC's be just as fast after the big request is long-gone (reclaimed) as they were before the big request?
Thanks.
On 22 October 2012 17:27, Chris Muller asqueaker@gmail.com wrote:
Eliot is referring to Ian Piumarta's explanation in sqUnixMemory.c here:
/* Note: * * The code allows memory to be overallocated; i.e., the initial * block is reserved via mmap() and then the unused portion * munmap()ped from the top end. This is INHERENTLY DANGEROUS since * malloc() may randomly map new memory in the block we "reserved" * and subsequently unmap()ped. Enabling this causes crashes in * Croquet, which makes heavy use of the FFI and thus calls malloc() * all over the place. * * For this reason, overallocateMemory is DISABLED by default. * * The upshot of all this is that Squeak will claim (and hold on to) * ALL of the available virtual memory (or at least 75% of it) when * it starts up. If you can't live with that, use the -memory * option to allocate a fixed size heap. */See also the man page for sbrk for some related cautions about mixing malloc with lower level allocation/deallocation calls.
If the VM starts wth a block initially allocated for the object-memory, and extended as needed for subsequent allocation (String new: 4000000), but THEN reduced once not needed -- why would a *subsequent* malloc call grabbing memory from the part that was deallocated cause a crash? At first I thought the crash would occur after a subsequent extension of object-memory into memory that was malloc'd for another purpose (plugin) but.. that could happen on the _first_ time it is extended, couldn't it?
IOW:
I start out using 40MB for object memory.
0----------------------40MBPlugins make some malloc calls, which allocate somewhere just above..
0----------------------40MB-----+5MB(malloc'd)Now I need more allocation for object memory, so I'm trampling what the plugin malloc'd?
Sorry, I unfortunately do not know C at all, thanks for helping me understand the situation.
Chris, may I ask why you see this as a problem? Is there some practical situation in which you are running out of real system memory as a consequence of the memory mapping strategy?
I have a Squeak-based server where 99% of requests are small, but once a big request comes in, that server image jumps from 40MB to 400MB. The extra memory was only needed for 10 seconds but unfortunately sits there unused forever-after. So it's almost like a memory-leak where the only way to recover it is to bounce the server.
What would be worse is if the over-allocated image then affects GC performance. I'm still trying to figure out whether this is the case -- it seems like not thank goodness. Do you know for sure the answer to that? IOW should full GC's be just as fast after the big request is long-gone (reclaimed) as they were before the big request?
yes, don't worry about that.
Thanks.
squeak-dev@lists.squeakfoundation.org
-
Chris Muller -
David T. Lewis -
Igor Stasenko