Dynamic system memory use

Marcel Weiher marcel at metaobject.com
Fri Feb 1 19:01:23 UTC 2002


On Friday, February 1, 2002, at 10:22 , John M McIntosh wrote:

>> Any particular reason for using mmap() and not just simply malloc()? 
>> It will have the same semantics of just allocating address-space.
>
> I started with mmap because of the linux work. Also I'm not sure if 
> malloc would reserve all 512mb right away for me.

It would.  It has to, since it guarantees to return a chunk of 
sequential memory.  With such a large allocation, you will get 
zero-filled VM-pages.

>>
>> One reason for mmap() would be that you can use it to create a private 
>> swapfile, which can be significantly better than going through the 
>> system swapfiles in some circumstances.
>>
>> Also, I started implementing the grow/shrink primitives, but then 
>> couldn't figure out how such an implementation differs from simply 
>> starting Squeak with -memory 512, in the presence of a lazy VM 
>> subsystem.
>
> Good question, when does the grow happen? Versus if you have 500MB of 
> free space.

I am not quite sure what you mean here.  There is no real growth, as in 
realloc() or the like.  As soon as you touch a page (read or write, it 
seems, although it could really delay until a write), the kernel will 
provide real memory for it, but no sooner.

>>
>>> Technically I could do an madvise with MADV_FREE when we shrink to 
>>> state the excess pages
>>> aren't needed and can be freed but maybe we should see what the 
>>> behavior is first. Mmm
>>> actually doing the MADV_FREE might be a good idea, we'll see for 
>>> 3.2.3, I suspect it will make the VM subsystem more happy, actually 
>>> some reading shows it will make it *very* happy.
>>
>> You have to be careful with these.  Some are not supported, some don't 
>> really work and others give you kernel panics. Apple has some 
>> bug-reports from me...
>
> I'd be interested to know what they are, and if you can issue MADV_FREE 
> without fear, since it seems the proper way to do things.

The answer is:  I am not sure.  What I found is that lockups were 
triggered under some circumstances that depended on external/variable 
factors such as total VM subsystem load etc.  While I can describe the 
particular circumstances that triggered it for me, I am not at all sure 
wether there aren't some other circumstances that would trigger similar 
or other unintended behaviors.  I simply wouldn't mess with this stuff 
right now.

Anyway, the real proper way to do this on Mach would be a user-level 
pager.  Alas, the user-level pager facility (EMMI, External Memory 
Manager Interface) is 'resting' in current Mac OS X.  Apparently, some 
of the low-level mechanisms it relies on are being revamped, and I was 
assured that it would be coming back by one of the kernel engineers.

In the meantime, I think just setting a large memory size and letting 
the VM subsystem handle it is probably fully adequate.

Marcel




More information about the Squeak-dev mailing list