[Vm-dev] Re: [squeak-dev] To a terabyte and... beyond

[David T. Lewis]

On Thu, Oct 28, 2010 at 09:25:18PM -0700, Dan Ingalls wrote:
> Hi Guys ;-)
> 
> Anyone have experience with running super huge Squeaks?  I have the 
> chance to use an absurdly big machine.  I'd appreciate advice 
> (offline unless you think it would be generally useful) regarding the 
> relative stability of, say, linux and windoze builds of 64-bit 
> Squeaks known (or thought) to run with over a terabyte of objects, 
> and also where there may be some pre-built 64-bit images (sorry if 
> this is in an obvious place).  Finally (since I've mainly used a Mac 
> for Squeak) I guess I'll have to know how to tell it how big to be, 
> or would it grow gracefully if I just said "(String new: 
> 1000000000000) size" in a normal image?

Hi Dan, 

I've made a few updates today to VMMaker and the unix platform code that
should make it easier to experiment with large object memories. These
consist of a few type declaration fixes and updates to permit larger
expansion of object memory and allocation of large objects when word
size is 8 and host platform is 64-bit.

I put a copy of a  working Squeak 3.8 based 64-bit image, along with VMMaker
and platform sources, at http://squeakvm.org/squeak64.

The image is at http://squeakvm.org/squeak64/sq64-20101119-dtl.zip. The
image along with a copy of complete platform sources and generated C code
from VMMaker is at http://squeakvm.org/squeak64/sq64-20101119-dtl.zip.

The VMMaker in this image is derived from http://www.squeaksource.com/VMMaker/VMMaker-dtl.200.mcz.
I manually removed various things that are not compatible with Squeak 3.8
(pragma declarations and various things related to the blockclosure updates)
in order to make it loadable in Squeak 3.8. There may be some errors but
I think this is functionally equivalent to VMMaker-dtl.200.mcz. I am not
yet able to trace a current Squeak trunk image (the SystemTrace rcan be
found at http://www.squeaksource.com/SystemTracing), which would surely
be a very good thing to do at this point.

The PC that I used has 8MB of real memory, and is running Linux. On this
box, I am able to create an image of about 7MB with no trouble, and to
save an apparently valid snapshot to disk (although I cannot reload
anything bigger than a few GB from disk on this system).

The garbage collector seems to still work under these conditions, although
I'm not really stressing it since I'm only allocating a small number of
very large objects. John pointed out a few likely problem areas here; I'm
sure there will be quite a few.

Objects (e.g. a String) of several GB can be allocated and seem to function
correctly, but do not display properly in an inspector.

Hopefully this will allow you to do some experimenting on a box with
very large memory. A terabyte seems like a bit of a stretch for a software
system that started out on 16 bit hardware, but we're only a few orders of
magnitude away so who is to say it can't be done?

Attached is a screen shot of the 64-bit image that you and Ian originally
did, now bloated out to about 7GB in size after allocating some gigabyte-sized
strings.

Please keep us posted if you break any interesting boundaries :)

Dave

-------------- next part --------------
A non-text attachment was scrubbed...
Name: world.png
Type: image/png
Size: 60673 bytes
Desc: not available
Url : http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20101102/22536fb6/world-0001.png