...stinks. I'm talking about sqAllocateMemory. Here's the base definition from platforms/Cross/vm/sq.h:
#define sqAllocateMemory(minHeapSize, desiredHeapSize) malloc(desiredHeapSize)
The problem here is that there's no obvious way for the client to know how much memory is returned. The signature implies it is somewhere between minHeapSize & desiredHeapSize inclusive (or null, if allocation failed). But how do you tell? Well, one could always ask to grow memory by 0 and see what you get back, except for the small chicken-and-egg problem that sqGrowMemory:By: and sqShrinkMemory:By: require the ammount of memory allocated as an argument:
#define sqGrowMemoryBy(oldLimit, delta) oldLimit #define sqShrinkMemoryBy(oldLimit, delta) oldLimit
So one has to go to extraordinary lengths that are completely non-obvious in the client code to actually pass-back the ammount allocated. Here's a client in readImageFromFile:
"allocate a contiguous block of memory for the Squeak heap" memory *:=* self cCode: 'sqAllocateMemory(minimumMemory, heapSize)' . memory = nil ifTrue: [self insufficientMemoryAvailableError].
memStart *:=* self startOfMemory. self setMemoryLimit: (memStart + heapSize) - 24. "decrease memoryLimit a tad for safety" self setEndOfMemory: memStart + dataSize.
and here's how one gets around the absence of a "how much was allocated" parameter, from platforms/Mac OS/vm/sqPlatformSpecifc.h
#undef sqAllocateMemory
usqInt sqAllocateMemoryMac(sqInt minHeapSize, sqInt *desiredHeapSize);
#define sqAllocateMemory(x,y) sqAllocateMemoryMac(x,&y);
Ingenious (seriously; John this is a very clever working around of the restriction). Totally non-obvious at the client site, but it still gets the job done without requiring the client to change. So this costs too much to maintain (a.k.a. puts too much strain on my rodent brain).
We could go with an explicit out parameter, as in
"allocate a contiguous block of memory for the Squeak heap" memory *:=* self cCode: 'sqAllocateMemory(minimumMemory, heapSize, &allocatedSize)'. memory = nil ifTrue: [self insufficientMemoryAvailableError].
memStart *:=* self startOfMemory. self setMemoryLimit: (memStart + allocatedSize) - 24. "decrease memoryLimit a tad for safety" self setEndOfMemory: memStart + dataSize.
but out parameters are not supported by Smalltalk and the VM is a Smalltalk program goddammit.
So I want to canvas opinions on the following simplified proposal, which is to drop the oldLimit parameter from sqGrowMemoryBy, discard sqShrinkMemoryBy (it is superfluous given sqGrowMemoryBy can take a signed argument) and use self sqGrowMemoryBy: 0 to determine how much memory has been allocated:
"allocate a contiguous block of memory for the Squeak heap" memory *:=* self sqAllocateAtLeast: minimumMemory AtMostMemory: heapSize. memory = nil ifTrue: [self insufficientMemoryAvailableError]. heapSize *:=* self sqGrowMemoryBy: 0. memStart *:=* self startOfMemory. self setMemoryLimit: (memStart + heapSize) - 24. "decrease memoryLimit a tad for safety" self setEndOfMemory: memStart + dataSize.
with the obvious default implementations in e.g. platforms/Cross/vm/sqMemory.c
static unsigned long memoryLimit = 0;
void * sqAllocateAtLeastAtMostMemory(unsigned long committed, unsigned long reserved) { void *mem; if ((mem = malloc(reserved)) ~= 0) memoryLimit = reserved; else if ((mem = malloc(committed)) ~= 0) memoryLimit = committed; return mem; }
unsigned long sqGrowMemoryBy(sqInt delta) { return memoryLimit; }
and everywhere in the VM which expects the old sqGrowMemoryBy to answer a pointer to the limit must instead add the result to the base of allocated memory; i.e.
limit *:=* self sqGrowMemory: memoryLimit By: delta. limit = memoryLimit ifFalse: [self setMemoryLimit: limit - 24. "remove a tad for safety"
self initializeMemoryFirstFree: freeBlock]
is rewritten as
limit *:=* memoryLimit + (self sqGrowMemoryBy: delta). limit = memoryLimit ifFalse: [self setMemoryLimit: limit - 24. "remove a tad for safety"
self initializeMemoryFirstFree: freeBlock] Opinions?
best Eliot
On 24-Jul-09, at 3:42 PM, Eliot Miranda wrote:
...stinks.
I'm talking about sqAllocateMemory. Here's the base definition from platforms/Cross/vm/sq.h:
#define sqAllocateMemory(minHeapSize, desiredHeapSize) malloc(desiredHeapSize)
The problem here is that there's no obvious way for the client to know how much memory is returned. The signature implies it is somewhere between minHeapSize & desiredHeapSize inclusive (or null, if allocation failed). But how do you tell? Well, one could always ask to grow memory by 0 and see what you get back, except for the small chicken-and-egg problem that sqGrowMemory:By: and sqShrinkMemory:By: require the ammount of memory allocated as an argument:
#define sqGrowMemoryBy(oldLimit, delta) oldLimit #define sqShrinkMemoryBy(oldLimit, delta) oldLimit
So one has to go to extraordinary lengths that are completely non- obvious in the client code to actually pass-back the ammount allocated. Here's a client in readImageFromFile:
"allocate a contiguous block of memory for the Squeak heap" memory := self cCode: 'sqAllocateMemory(minimumMemory,heapSize)'. memory = nil ifTrue: [self insufficientMemoryAvailableError].
memStart := self startOfMemory. self setMemoryLimit: (memStart + heapSize) - 24. "decreasememoryLimit a tad for safety" self setEndOfMemory: memStart + dataSize.
Somehow that looks dated? Since it now reads
"allocate a contiguous block of memory for the Squeak heap" memory := self allocateMemory: heapSize minimum: minimumMemory imageFile: f headerSize: headerSize. memory = nil ifTrue: [self insufficientMemoryAvailableError].
which turns into
memory = allocateMemoryMinimumImageFileHeaderSize(heapSize, minimumMemory, f, headerSize); if (memory == null) { insufficientMemoryAvailableError(); }
which is #define allocateMemoryMinimumImageFileHeaderSize(heapSize, minimumMemory, fileStream, headerSize) \ sqAllocateMemory(minimumMemory, heapSize) #endif
but on iPhone and mac is
#define allocateMemoryMinimumImageFileHeaderSize(heapSize, minimumMemory, fileStream, headerSize) \ sqAllocateMemoryMac(heapSize, minimumMemory, fileStream, headerSize)
and
sqAllocateMemoryMac
On the iPhone let's you vmap in the image from offset 500*1024*1024 plus header size to the size of the image file rounded up 4K pages. After that the free space is mmap anonymous upto the total heapsize, we ignore minimumMemory. For WikiServer a 10MB image, then 6MB gets paged in from flash, 4MB is not touched. The sqImageFileReadEntireImage does nothing.
On the macintosh the total heap size is mmapped anonymously at the 500*1024*1024 boundary plus header size, the sqImageFileReadEntireImage then reads the image file into the mmap region. I note that I had the same code here from the iPhone but it was discovered there is a bug with mmapped files being read from NFS disks so the feature was made optional. It does btw save a few 100 ms at startup time on slower machines (500 Mhz) But the macintosh virtual memory system does read all the pages from the file into RAM, versus the iPhone which does not. -- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================
Hi John, privately...
On Fri, Jul 24, 2009 at 4:02 PM, John M McIntosh < johnmci@smalltalkconsulting.com> wrote:
On 24-Jul-09, at 3:42 PM, Eliot Miranda wrote:
...stinks.
I'm talking about sqAllocateMemory. Here's the base definition from platforms/Cross/vm/sq.h:
#define sqAllocateMemory(minHeapSize, desiredHeapSize) malloc(desiredHeapSize)
The problem here is that there's no obvious way for the client to know how much memory is returned. The signature implies it is somewhere between minHeapSize & desiredHeapSize inclusive (or null, if allocation failed). But how do you tell? Well, one could always ask to grow memory by 0 and see what you get back, except for the small chicken-and-egg problem that sqGrowMemory:By: and sqShrinkMemory:By: require the ammount of memory allocated as an argument:
#define sqGrowMemoryBy(oldLimit, delta) oldLimit #define sqShrinkMemoryBy(oldLimit, delta) oldLimit
So one has to go to extraordinary lengths that are completely non-obvious in the client code to actually pass-back the ammount allocated. Here's a client in readImageFromFile:
"allocate a contiguous block of memory for the Squeak heap" memory := self cCode: 'sqAllocateMemory(minimumMemory, heapSize)'. memory = nil ifTrue: [self insufficientMemoryAvailableError]. memStart := self startOfMemory. self setMemoryLimit: (memStart + heapSize) - 24. "decreasememoryLimit a tad for safety" self setEndOfMemory: memStart + dataSize.
Somehow that looks dated? Since it now reads
"allocate a contiguous block of memory for the Squeak heap" memory := self allocateMemory: heapSize minimum: minimumMemory imageFile: f headerSize: headerSize. memory = nil ifTrue: [self insufficientMemoryAvailableError].which turns into
memory = allocateMemoryMinimumImageFileHeaderSize(heapSize,minimumMemory, f, headerSize); if (memory == null) { insufficientMemoryAvailableError(); }
which is #define allocateMemoryMinimumImageFileHeaderSize(heapSize, minimumMemory, fileStream, headerSize) \ sqAllocateMemory(minimumMemory, heapSize) #endif
Yes, but that's beside the point. The basic issue still remains. I'd still like your opinion on the question asked. Would you be willing to answer again? Don't assume that all platforms will reserve the entire memory. Yes, all the platforms we use here do but a bare metal port might still use malloc.
but on iPhone and mac is
#define allocateMemoryMinimumImageFileHeaderSize(heapSize, minimumMemory, fileStream, headerSize) \ sqAllocateMemoryMac(heapSize, minimumMemory, fileStream, headerSize)
and
sqAllocateMemoryMac
On the iPhone let's you vmap in the image from offset 500*1024*1024 plus header size to the size of the image file rounded up 4K pages. After that the free space is mmap anonymous upto the total heapsize, we ignore minimumMemory. For WikiServer a 10MB image, then 6MB gets paged in from flash, 4MB is not touched. The sqImageFileReadEntireImage does nothing.
On the macintosh the total heap size is mmapped anonymously at the 500*1024*1024 boundary plus header size, the sqImageFileReadEntireImage then reads the image file into the mmap region. I note that I had the same code here from the iPhone but it was discovered there is a bug with mmapped files being read from NFS disks so the feature was made optional. It does btw save a few 100 ms at startup time on slower machines (500 Mhz) But the macintosh virtual memory system does read all the pages from the file into RAM, versus the iPhone which does not.
--
John M. McIntosh johnmci@smalltalkconsulting.com Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ===========================================================================
On Fri, Jul 24, 2009 at 4:59 PM, Eliot Miranda eliot.miranda@gmail.comwrote:
Hi John, privately...
so much for that. gmail seems hopelessly inconsistent form day to day as to whether Reply does reply to all or reply to sender. Or is it my advancing years??
On Fri, Jul 24, 2009 at 4:02 PM, John M McIntosh < johnmci@smalltalkconsulting.com> wrote:
On 24-Jul-09, at 3:42 PM, Eliot Miranda wrote:
...stinks.
I'm talking about sqAllocateMemory. Here's the base definition from platforms/Cross/vm/sq.h:
#define sqAllocateMemory(minHeapSize, desiredHeapSize) malloc(desiredHeapSize)
The problem here is that there's no obvious way for the client to know how much memory is returned. The signature implies it is somewhere between minHeapSize & desiredHeapSize inclusive (or null, if allocation failed). But how do you tell? Well, one could always ask to grow memory by 0 and see what you get back, except for the small chicken-and-egg problem that sqGrowMemory:By: and sqShrinkMemory:By: require the ammount of memory allocated as an argument:
#define sqGrowMemoryBy(oldLimit, delta) oldLimit #define sqShrinkMemoryBy(oldLimit, delta) oldLimit
So one has to go to extraordinary lengths that are completely non-obvious in the client code to actually pass-back the ammount allocated. Here's a client in readImageFromFile:
"allocate a contiguous block of memory for the Squeak heap" memory := self cCode: 'sqAllocateMemory(minimumMemory, heapSize)'. memory = nil ifTrue: [self insufficientMemoryAvailableError]. memStart := self startOfMemory. self setMemoryLimit: (memStart + heapSize) - 24. "decreasememoryLimit a tad for safety" self setEndOfMemory: memStart + dataSize.
Somehow that looks dated? Since it now reads
"allocate a contiguous block of memory for the Squeak heap" memory := self allocateMemory: heapSize minimum: minimumMemory imageFile: f headerSize: headerSize. memory = nil ifTrue: [self insufficientMemoryAvailableError].which turns into
memory = allocateMemoryMinimumImageFileHeaderSize(heapSize,minimumMemory, f, headerSize); if (memory == null) { insufficientMemoryAvailableError(); }
which is #define allocateMemoryMinimumImageFileHeaderSize(heapSize, minimumMemory, fileStream, headerSize) \ sqAllocateMemory(minimumMemory, heapSize) #endif
Yes, but that's beside the point. The basic issue still remains. I'd still like your opinion on the question asked. Would you be willing to answer again? Don't assume that all platforms will reserve the entire memory. Yes, all the platforms we use here do but a bare metal port might still use malloc.
but on iPhone and mac is
#define allocateMemoryMinimumImageFileHeaderSize(heapSize, minimumMemory, fileStream, headerSize) \ sqAllocateMemoryMac(heapSize, minimumMemory, fileStream, headerSize)
and
sqAllocateMemoryMac
On the iPhone let's you vmap in the image from offset 500*1024*1024 plus header size to the size of the image file rounded up 4K pages. After that the free space is mmap anonymous upto the total heapsize, we ignore minimumMemory. For WikiServer a 10MB image, then 6MB gets paged in from flash, 4MB is not touched. The sqImageFileReadEntireImage does nothing.
On the macintosh the total heap size is mmapped anonymously at the 500*1024*1024 boundary plus header size, the sqImageFileReadEntireImage then reads the image file into the mmap region. I note that I had the same code here from the iPhone but it was discovered there is a bug with mmapped files being read from NFS disks so the feature was made optional. It does btw save a few 100 ms at startup time on slower machines (500 Mhz) But the macintosh virtual memory system does read all the pages from the file into RAM, versus the iPhone which does not. --
=========================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com
===========================================================================
so much for that. gmail seems hopelessly inconsistent form day to day as to whether Reply does reply to all or reply to sender. Or is it my advancing years??
Eyesight.
Well I did phone Eliot, and said.
Really the whole issue is the memory allocation, read into memory, grow shrink can be re-done, as you noticed it started simple with a malloc() then grew into something more complicated and twisted trying not to modify code in VMMaker, so you could build with old src code yet get newer memory allocation logic.
In the refactor consider
(a) we allocate a certain amount of memory, this is the file size plus some ?unknown? small amount? Upto some maximum. The maximum in the past can be a fixed amount, or even a value in additional to the file size, or percent of file size etc. Lately it seems to be a fixed upper limit. This is also then altered by the ability of the operating system to give you the memory, then we take some min-heap size as a drop dead value, which in my opinion is a wild ass guess. So the value becomes something between the min heap size and the requested value, again I don't think the developer has a clue here if it was acceptable.
(b) Allocating the memory may or may not require the use of the file since it could be needed for a mmap
(c) Memory can grow or shrink, few platforms offer fail safe shrink logic.
(d) After the memory is allocated the file has to be read into it, or paged into it, or nothing happens if the file was mmapped in via (a)
-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================
On Fri, Jul 24, 2009 at 05:23:51PM -0700, John M McIntosh wrote:
Really the whole issue is the memory allocation, read into memory, grow shrink can be re-done, as you noticed it started simple with a malloc() then grew into something more complicated and twisted trying not to modify code in VMMaker, so you could build with old src code yet get newer memory allocation logic.
I'm somewhat responsible for the complicated and twisted part of this, which could easily be extended to something only slightly more complicated and twisted if the only motivation is to answer the size of the allocated memory as a by-product of the allocation request.
But what is the motivation for the question? Is the concern just to be able to release memory back to the platform, or is there a more general need for a memory allocation interface to support future object memory designs?
Dave
2009/7/25 David T. Lewis lewis@mail.msen.com:
On Fri, Jul 24, 2009 at 05:23:51PM -0700, John M McIntosh wrote:
Really the whole issue is the memory allocation, read into memory, grow shrink can be re-done, as you noticed it started simple with a malloc() then grew into something more complicated and twisted trying not to modify code in VMMaker, so you could build with old src code yet get newer memory allocation logic.
I'm somewhat responsible for the complicated and twisted part of this, which could easily be extended to something only slightly more complicated and twisted if the only motivation is to answer the size of the allocated memory as a by-product of the allocation request.
But what is the motivation for the question? Is the concern just to be able to release memory back to the platform, or is there a more general need for a memory allocation interface to support future object memory designs?
I remember i wrote couple of words about memory allocation and its limitations in current implementation. Mainly my concerns are: - be able to reallocate the memory (shrinking/growing), not only allocate/deallocate
my idea is to use a _single_ memory allocation function:
void * sqRealloc(void * memory, uint size);
- if memory argument is nil -> then function should allocate a new block. - if memory is not nil -> then fuction should try allocate enough space to fit the requested size without moving the memory but if memory block has to be moved during reallocation - then returned pointer can be not the same as what is passed to function.
- if you pass size= 0 it should deallocate the memory block
the VM should take care if memory block were moved as a result of reallocation (like relocating object memory , what it already does when loading new image).
What could be simpler? :)
Dave
Also all the unix based systems (is there any other kind) just fiddle with the sqGrowMemoryBy logic by moving a pointer about indicating where the end of memory is since they all allocate the entire heapsize via mmap at startup time. Since all those unix systems do lazy allocation of memory the fact you ask for 1GB but use 20MB doesn't *really*(1) matter.
In the past some VM developers have be *confused* about what that does and implemented logic to remap the image memory based on grow/shrink, but later discover fascinating bugs with how the remap works, or the expensive of doing so.
(1) Well it does in terms of memory used to track that 1GB of virtual memory
On 24-Jul-09, at 3:42 PM, Eliot Miranda wrote:
unsigned long sqGrowMemoryBy(sqInt delta) { return memoryLimit; }
and everywhere in the VM which expects the old sqGrowMemoryBy to answer a pointer to the limit must instead add the result to the base of allocated memory; i.e.
limit := self sqGrowMemory: memoryLimit By: delta. limit = memoryLimit ifFalse: [self setMemoryLimit: limit - 24. "remove a tad forsafety" self initializeMemoryFirstFree: freeBlock]
is rewritten as
limit := memoryLimit + (self sqGrowMemoryBy: delta). limit = memoryLimit ifFalse: [self setMemoryLimit: limit - 24. "remove a tad forsafety" self initializeMemoryFirstFree: freeBlock] Opinions?
best Eliot
-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================
John M McIntosh wrote:
Also all the unix based systems (is there any other kind)
Yes.
just fiddle with the sqGrowMemoryBy logic by moving a pointer about indicating where the end of memory is since they all allocate the entire heapsize via mmap at startup time. Since all those unix systems do lazy allocation of memory the fact you ask for 1GB but use 20MB doesn't *really*(1) matter.
The real concern is how to release memory. It's fine to use a spike of a few hundred megs if you need it temporarily but the inability to give it back to the OS really sucks. One of the results is that you can't average memory allocation across multiple images but have to assume worst-case consumption for any single instance.
Cheers, - Andreas
vm-dev@lists.squeakfoundation.org
-
Andreas Raab -
David T. Lewis -
Eliot Miranda -
Igor Stasenko -
John M McIntosh