[Vm-dev] Fwd: Nix package for Pharo flavor of opensmalltalk-vm

Sun Apr 23 17:22:52 UTC 2017

Hi Subbu,

> On Apr 23, 2017, at 6:25 AM, K K Subbu <kksubbu.ml at gmail.com> wrote:
> 
>> On Thursday 20 April 2017 04:13 PM, Luke Gorrie wrote:
>> Pharo upstream are distributing releases as binary vm+image pairs with
>> some third party libraries included. The most faithful way to package
>> this for nix would be to take these binaries and make them work (using
>> patchelf to fix up the shared library paths.) This is how people package
>> other binary software for nix e.g. Skype.
> 
> The vm+image is like chicken+egg ;-). Separated across time, they have a recursive dependency between them. VM Maker app run on an existing image (Pharo 5 or Squeak 5) to generate a new VM (e.g. Spur) and that in turn is used to evolve the next image release (e.g. Squeak 6, Pharo 6). VM is machine-specific code while Image an object graph on disk.
> 
> This is analogous to linux+rootfs except that linux can handle multiple filesystems while our VM can only deal with one format at this time.

I think you're missing one, possibly two things more, and that is the object model, and the execution model.  Since 2008 I have been changing these to be able to increase performance and functionality.

The execution model concerns things like
- the representation of classes (their format word, their methodDictionary layout)
- the layout of a CompiledMethod header word (its first slot, objectAt: 1), which determines the max number of literals, argument count, temporary count, and range of available primitives
- how blocks work (whether they are non-reentrant as in the blue book, or fully reentrant closures
- the bytecode set, which determines things like the max number of instance variables in an instance, the max number of literals in a method, the max jump displacement
- how blocks and methods are related, whether block bytecode is embedded in the home method's bytecode or in an independent CompiledBlock object

The object model is the representation of objects in the heap, and concerns things like
- what are the layouts and different kinds of object headers, which determine things like the identityHash range, whether the system can support ephemerons, whether there are 8, 16, 32 and 64-bit non-pointer collections
- what the tag organization is, which determines whether there are immediate characters or immediate floats
- whether the heap can be grown by segments or not

The first major change I introduced was the block model to introduce fully reentrant closures.  Apart from being preferrable in and of themselves they also, when married with avoid representation for accessing closed over variables (see the bytecode for inject:into: and the creation of the indirection vector to hold nextValue, fully explained on my blog), enables the VM to efficiently and transparently map contexts to stack frames in the VM.  This is the StackInterpreter and Cog V3 VMs first released around 2010.

These systems share the object representation of the older Squeak 3.9 and earlier versions, but won't run older images because they depend on the older blue book non-reentrant block model.  

Note that it is a non-trivial change, affects my not just the VM, the bytecode set and the compiler, but the debugger, object (de)serialization, and some client code that depended on old block semantics.

Maintaining backward compatibility here would have hobbled the VM and severely limited the potential performance increase from context-to-stack mapping that gave us the Cog VM which is around 7x faster than the Squeak 3.9 interpreter.  So the break with backward compatibility was done consciously and with good reason.

The other change I introduced was Spur, which provides a very different object representation that
- simplifies the object header to one or two 64-but words
- increases the identityHash width
- introduces immediate characters and in 64-bits immediate floats for the most used 1/8 of the double precision range
- allows the heap to grow in segments
- provides a full 64-bit implementation (with 61-bit SmallIntegers) that shares the object header format between 32 and 64 bits
- allows the VM to implement much more Instantiations code in machine code because it is simpler

Maintaining backward compatibility here is simply not possible.  The bootstrap is ling and complex.  It had to be done as a new release.  Spur is around 2x the speed of the Cog V3 VM (some things are unchanged, some things, like widestring access are much faster).  

I took the opportunity to make an execution model change, moving the primitive field from the method header into the bytecode, to increase the number of literals to 32k. This also needs a bytecode change to enable 32k literaks and that evolution is taking place now with the Sista/Scorch development which will provide adaptive optimization via speculative inlining.

Note that while the conversion between 32-bit and 64-bit Spur images is automatic it is slow; minutes not seconds.  Not something one would want to inflict on people on startup, especially not as the first experience of using the system, hence separate 32-bit and 64-bit images.

So the break with backward compatibility, and the split between 32 and 64 bit images was done consciously and with good reason.

Now once the dust has settled the VM attempts to provide backward compatibility with any image that has the right execution model and object model.  One should always be able to use a Spur VM with any older Spur image, and any CogV3 CM with any older 4.x image.  Newer VMs may introduce new functionality that new images may depend on, but older images should continue to work.

To simplify this conceptually we boil this down to
- the newest vm for a major release is  backwards compatible with older images for that last major release

> In Linux, filesystem modules translates objects on disk to object graph (tree) in memory and the rest of the kernel refer to these objects through a VFS namespace, so they are oblivious to the physical layout on disk. This indirection is nicely brought out in :
> 
> https://www.dmst.aueb.gr/dds/pubs/inbook/beautiful_code/html/Spi07g.html
> 
> Linux itself is split into the kernel+initramfs, but there is no such equivalent in our VM. We could introduce indirection by building a bootstrap image into the VM itself (perhaps as a special plugin) and have the vm load it initially. This image can then probe user-given image files, match it with the right loader plugins and then switch to it.

But the information that differentiates between images is encoded in the image header version number and that can be accessed via od or hexdump et al and so can be done with a shell script.

I hope you can see that the differences between systems are more than between file system formats; it's not just a case of having the right driver.  Because the system is self defining there is much more of a circular dependency between vm and image.

I'm sorry that the complexity is inconvenient.  I do hope that things will settle down over the next few years after Sista/scorch is released.

> 
> Regards .. Subbu
> "All problems in computer science can be solved by another level of indirection" .. David Wheeler