Dynamic system memory use

John M McIntosh johnmci at smalltalkconsulting.com
Fri Feb 1 07:59:49 UTC 2002


>Tim Rowledge <tim at sumeru.stanford.edu> wrote:
>>  Squeak can increase the memory it uses if the OS supports a suitable
>>  method of extending the original allocation - or potentially making a
>>  new larger allocation and copying the entire object memory across and
>>  swizzling all the pointers. See the macros
>>      #define sqGrowMemoryBy(oldLimit, delta) oldLimit
>>      #define sqShrinkMemoryBy(oldLimit, delta) oldLimit
>>      #define sqMemoryExtraBytesLeft(includingSwap) 0
>>  in sq.h and their uses.
>
>Unfortunately, these functions are hard to implement.  They expect the
>heap to be reallocated *in place*, and more importantly, they expect the
>heap to stay as one contiguous segment.  realloc() could probably be
>made to work, but it's still going to be difficult on the underlying OS,
>simply because realloc-ing hundreds of megs will make the 4 gigs of
>32-bit addressable space start running out!

Actually because of real pointers and how the VM works, it's only 
2GB, the first 2.

>
>I don't know what is being done on the Windows and MacOS ports to
>support these.  On Unix, all I could think of is to allocate a huge
>amount of space using mmap, and then not touch it until it's needed.
>This works for allocation.  For deallocation, munmap can be used,
>followed by an immediate mmap.  Unfortunately, there is no real
>guarantee that the second mmap is going to work, which would mean the
>system suddenly has a very small amount of free heap space!
>

For OS-X I mmap as anonymous 512MB. I then move the end of memory 
location within those 512MB. If the memory isn't used it never gets 
allocated. If the memory gets used, and is paged out to disk, *and* 
we shrink the image size, well I just ignore it and have the virtual 
memory manager leave those inactive pages sit on the disk. If we 
grow, then the cost is mapping those pages in to be overwritten, but 
the cost is no different than taking a page fault to get more memory 
in a malloc situation.  (but see my note about madvise)

So mmap and forget about an ummap and mmap.

Lets see..

PhysMem:  60.2M wired, 78.6M active,  412M inactive,  551M used,  217M free
VM: 3.03G + 55.3M   7847(0) pageins, 1(0) pageouts
PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
422 Squeak VM    0.8%  0:35.59   2    71   100  23.3M  8.53M  26.5M   571M

	but I only have one 80,000,000 byte swap file in /var/vm

after allocating 400K bytearray, with another squeak vm, so it's not 
quite before/after

PhysMem:  64.0M wired, 55.7M active,  639M inactive,  759M used, 9.39M free
VM: 3.21G + 50.5M   16138(13) pageins, 5904(0) pageouts
PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
1235 Squeak VM    1.5%  0:14.43   3    77   119   421M  8.02M   424M   573M

Now I have two 80,000,000 byte swap file in /var/vm

Lets Free and GC

memory			441,367,964 bytes
	old			17,540,780 bytes (4.0%)
	young		419,621,884 bytes (95.1%)
	used		437,162,664 bytes (99.0%)
	free		4,205,300 bytes (1.0%)
GCs				395 (482ms between GCs)
	full			2 totalling 1,432ms (1.0% uptime), avg 716.0ms
	incr		393 totalling 685ms (0.0% uptime), avg 2.0ms
	tenures		0
goes to

memory			21,753,888 bytes
	old			17,559,608 bytes (80.7%)
	young		81,152 bytes (0.4%)
	used		17,640,760 bytes (81.1%)
	free		4,113,128 bytes (18.9%)
GCs				505 (424ms between GCs)
	full			3 totalling 2,123ms (1.0% uptime), avg 708.0ms
	incr		502 totalling 900ms (0.0% uptime), avg 2.0ms
	tenures		0

MemRegions: num = 3635, resident =  498M + 6.66M private, 57.3M shared
PhysMem:  64.0M wired, 58.2M active,  634M inactive,  756M used, 12.1M free
VM: 3.21G + 50.5M   16150(1) pageins, 6433(0) pageouts

   PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD  RSIZE  VSIZE
  1235 Squeak VM    9.3%  0:36.43   2    75   117   421M  8.00M   424M   573M

Technically I could do an madvise with MADV_FREE when we shrink to 
state the excess pages
aren't needed and can be freed but maybe we should see what the 
behavior is first. Mmm
actually doing the MADV_FREE might be a good idea, we'll see for 
3.2.3, I suspect it will make the VM subsystem more happy, actually 
some reading shows it will make it *very* happy.

Note somehow I miss my Linux box, so I could go do a man madvise and 
see if that is supported.
Well I can hunt on the internet but it's not quite the same (which 
version is that for?)
I *think* linux is MADV_DONTNEED, but I'm not sure exactly what it 
does (conflicting stories).


-- 
--
===========================================================================
John M. McIntosh <johnmci at smalltalkconsulting.com> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================



More information about the Squeak-dev mailing list