AW: Dynamic system memory use

Marcel Weiher marcel at metaobject.com
Fri Feb 1 21:41:23 UTC 2002


On Friday, February 1, 2002, at 09:29 , Andreas Raab wrote:

>> Also, I started implementing the grow/shrink primitives, but then
>> couldn't figure out how such an implementation differs from simply
>> starting Squeak with -memory 512, in the presence of a lazy
>> VM subsystem.
>
> It differs in various ways.

Really?

>  First of all, let me repeat what's actually
> the expected behavior of any implementation for dynamic memory
> management functions:
>
> On startup we _reserve_ an (arbitrarily high) amount of address space.

This is exactly what doing a large malloc() does on Mach.

> The only reason we do this is to prevent the stupid C allocator from
> giving away memory with addresses in the specified region so that Squeak
> is guarantueed to get a linear chunk if requested.

Exactly.

>  At this point, we
> don't take a single byte of the memory, all we do is _reserving address
> space_.

This is exactly what doing a large malloc() does on Mach.

>  When we call one of the shrink/grow functions, then we actually
> request (continuous) portions of memory out of the reserved address
> space.

You get this on Mach (the 'grow') just by using the memory you allocated.

>  Now we need it and we know we need it. Growing may fail due to
> insufficient resources and that's fine; we can recover from such a
> failure quite nicely (e.g., you'll just get a low space warning).

I haven't had that sort of failure in about 8 years (not that much HD 
space on those Next boxes), but you probably get a signal that you could 
catch (or a message to your Mach exception port that you could also 
catch).

However, the system will have become quite unusable long before that.

> Compare that to your lazy VM subsystem: For one thing, your VM subsystem
> may require backing store for the entire reserved address space which is
> a significant waste of disk space (the current Windows VM reserves about
> 1.5GB of memory and I tend to run multiple images at the same time).

Mach doesn't do that.

> Secondly, if it does _not_ provide backing store, then your VM subsystem
> may randomly fail at some unspecified point of actually accessing the
> memory it promised to you.

Yes.  However, this is currently difficult to prevent on Mach.  One way 
that I have implemented is swapping to your own private swapfile.  
However, this means you have to pre-allocate this file for all the 
memory you want to use, which is very limiting, performane-wise.

>  Meaning that instead of a low space warning
> you'll get a core dump (if at all).

The easy way to prevent this is to set the -memory parameter to less 
than your available disk space.  Incidentally, your system will have 
ground to a halt thrashing long before you're out of disk-space.  Sorry, 
this is just a non-issue.

>  Thirdly, since the VM subsystem can
> hardly monitor whether we still write to pages of memory (which, after a
> shrink we do _not_) it is unable to release that portion of memory to
> the OS or other apps.

If they are not used, they well get paged out to disk and just sit 
unused in some swapfile.  If you really need the space back, quit and 
restart your image.  BFHD.

>  Fourthly, since Squeak has no knowledge about how
> much memory is actually available from the system without going to
> lengthy swapping,

Huh?

>  there is no way of optimizing large data set
> manipulations based on the amount of physical memory

Just set your -memory parameter to the amount of physical memory 
available.  What exactly is the problem?

>  (which can be a big
> advantage due to the unusually low localization of memory access
> patterns in OOP systems). So there's a _huge_ difference between a lazy
> VM subsystem and the behavior described above.

Not really.  The system is slightly more elegant, definitely.  But in 
practical terms I would wager that the differences are negligible.

> What I would recommend is looking at your common C allocator and find a
> way of reserving address space (is there a function to create a new heap
> in Linux?)

This function is called malloc() on Mac OS X. ;-)  (Actually, it is the 
behavior of the underlying vm_allocate(), but a large malloc() will 
essentially just call-through.)

>  and allocate/free continous portions from it (used to be
> called sbrk()).

No need to be condescending...

Anyway, here's what 'man sbrk' says:


BRK(2)                    System Programmer's Manual                    
BRK(2)

NAME
      brk, sbrk - change data segment size

SYNOPSIS
      #include <unistd.h>

      char *
      brk(const char *addr)

      char *
      sbrk(int incr)

DESCRIPTION
      The brk and sbrk functions are historical curiosities left over 
from ear-
      lier days before the advent of virtual memory management.

...

That's actually quite funny. ;-)  The explanation is that sbrk() and 
friends make assumptions about a specific layout of contiguous segments 
that is simply not valid on Mach.

Regards,

Marcel




More information about the Squeak-dev mailing list