[Vm-dev] Re: Only ever grows? (was: Re: [squeak-dev] how/where is memory returned to the OS?)

[David T. Lewis]

On Sun, Oct 21, 2012 at 09:35:15PM -0500, Chris Muller wrote:
> > Unfortunately there is no easy fix , which will fit all possible use scenarios.
> > As you can see from comments in code, this feature was disabled on purpose
> > for cases when your application works extensively with external
> > resources/libraries,
> > which using malloc() a lot.
> 
> You mean FFI?  Someone took the trouble to write this for the manpage.
>  My gut thinks FFI has been around longer than the releasing of memory
> feature -- not sure.
> 
>        -mmap size[mk]
>               requests that a variable heap of at most  size  bytes  be  allo???
>               cated.   (The  suffixes  are  as  described  for  the  '-memory'
>               option.)  squeak will initially allocate a heap  that  is  large
>               enough  to  hold the image, with a small amount of headroom.  If
>               at any time Squeak requires more memory for its image then addi???
>               tional space will be allocated dynamically.  Likewise, when mem???
>               ory is no longer needed it will deallocated and returned to  the
>               system.   The size argument places an upper limit on how big the
>               heap can grow in this fashion.  squeak uses a  dynamic  heap  by
>               default  with  the maximum size set to 75% of the available vir???
>               tual memory or 1 gigabyte, whichever is smaller.
> 
> Eliot, is the useMmap variable you referred to set by this argument?
> I tried launching my image with -mmap 700M but it still didn't seem to
> want to deallocate.
> 
> Same with the Classic VM.

Eliot is referring to Ian Piumarta's explanation in sqUnixMemory.c here:

    /* Note:
     * 
     *   The code allows memory to be overallocated; i.e., the initial
     *   block is reserved via mmap() and then the unused portion
     *   munmap()ped from the top end.  This is INHERENTLY DANGEROUS since
     *   malloc() may randomly map new memory in the block we "reserved"
     *   and subsequently unmap()ped.  Enabling this causes crashes in
     *   Croquet, which makes heavy use of the FFI and thus calls malloc()
     *   all over the place.
     *   
     *   For this reason, overallocateMemory is DISABLED by default.
     *   
     *   The upshot of all this is that Squeak will claim (and hold on to)
     *   ALL of the available virtual memory (or at least 75% of it) when
     *   it starts up.  If you can't live with that, use the -memory
     *   option to allocate a fixed size heap.
     */

See also the man page for sbrk for some related cautions about mixing malloc
with lower level allocation/deallocation calls.

Igor just posted a patch to let you override the default for the command
line, but you should be very sure that your application is not doing malloc
calls. Plugins (e.g. OSProcess) and FFI calls are entirely free to do this,
and you should assume that they will do so unless you positively know that
they do not.

Chris, may I ask why you see this as a problem? Is there some practical
situation in which you are running out of real system memory as a consequence
of the memory mapping strategy?

<OT>
I'll note for the record that Java VMs seem to adopt a much simpler strategy
for object memory management: Whenever you run low on memory, they crash. Yet
somehow people manage to build commercial systems on top of this. Go figure.
</OT>

Dave