[Vm-dev] (fwd) Re: 6502

Thu Apr 23 01:03:25 UTC 2009

On Wed, Apr 22, 2009 at 01:36:25PM -0300, Jecel Assumpcao Jr wrote:
>  
> Here is a discussion that I started on another list, but which we agreed
> that it would be better to have it here.
> 
> I should mention that I do see that version numbers are too linear to
> properly represent development that often branches in complex ways. I
> would personally prefer to use image version numbers as we do now (to
> indicate radical changes) and have some special in-image object to
> represent the features that are needed. It could be a well known array
> of symbols (or strings) like #(exceptions weakPointers exupery
> utf8input) so the VM could check that it can fully work with that image.
> But I think something like this is even less likely to be accepted than
> incrementing 6502...
> 

I suggest that we do the following to support future image format changes:

Treat the imageFormatVersion value as a bit map of size 32, derived from the
32-bit integer value saved in the image header. Bits in the bit map represent
image format requirements, e.g. if image sets bit 15 to indicate "needs object
header inversion", then VM can check bit 15 and decide whether it can support
this (whimsical) requirement.

As you suggest, a special in-image object should interpret the bit map to
represent the features and document their intent.

Given current usage (6502, 6502, 68000, and 68002) we have 10 bits currently
in use, and 22 available for future use. Reserve the high order bit for future
extention in case 32 bits is not enough. This leaves 21 bits for future image
format changes.

Make one change right now: The imageFormatVersion number for a 64-bit
image with closure bytecode support should be changed from 68002 to 68008.
This means that bit number 4 (one-base indexing) means "image requires
closure bytecode support" for both 32-bit and 64-bit image formats. The
change only effects people running 64-bit images with closures (I am quite
confident that this is a small number of people).

Policy: Anyone making a significant image format change that they wish
to be supported by standard VMs can register a bit in the imageFormatVersion
bit map. When we run out of bits, allocate a second 32 bit value, set the
extension bit, and carry on. If that's not enough, burn the disk packs ;)

How the imageFormatVersion value is used and modified:

Image responsibilities:
  - Image is responsible for knowing its imageFormatVersion.
  - Image is responsible for modifying its imageVersionNumber if new capabilities
    are required
  - Image is responsible for informing the VM of any change (note: An image
    with closure bytecodes does this, albeit indirectly).

VM responsibilities:
  - VM is responsible for reading imageFormatVersion from image header on image load
  - VM is responsible for determining if it can support the image format and for
    deciding what to do if it cannot support the image format
  - VM is responsible for saving the (possibly updated) imageFormatVersion
    on image save

Note: Eliot's implementation for closure support does all of the above.

For reference, here are the bits in the imageFormatVersion word that are
currently in use (1 means that the bit is used by one of the four current image
format numbers):

Bit 1 ==> 0
Bit 2 ==> 1
Bit 3 ==> 1
Bit 4 ==> 1	(the "closure bytecodes required" bit)
Bit 5 ==> 0
Bit 6 ==> 1
Bit 7 ==> 1
Bit 8 ==> 1
Bit 9 ==> 1
Bit 10 ==> 0
Bit 11 ==> 0
Bit 12 ==> 1
Bit 13 ==> 1
Bit 14 ==> 0
Bit 15 ==> 0
Bit 16 ==> 0
Bit 17 ==> 1
Bit 18 ==> 0
Bit 19 ==> 0
Bit 20 ==> 0
Bit 21 ==> 0
Bit 22 ==> 0
Bit 23 ==> 0
Bit 24 ==> 0
Bit 25 ==> 0
Bit 26 ==> 0
Bit 27 ==> 0
Bit 28 ==> 0
Bit 29 ==> 0
Bit 30 ==> 0
Bit 31 ==> 0
Bit 32 ==> 0	(reserve as extension bit)

- Dave

> ====forwarded====
> Hi Jecel -
> 
> I think this discussion is probably better done on either Squeak-dev or 
> VM-dev, so I'll only add one short comment. The current VMs do support 
> both the pre- and the post-closure format in precisely the way you are 
> describing. Eliot made it so that an image is being marked as closure 
> version if the closure creation bytecode is ever executed so that one 
> may load a pre-closure image, run the conversion script, and save it as 
> a post-closure version. In the future (i.e., when using the Stack VM) 
> the pre-closure format will no longer be supported but for the interim 
> period both are supported.
> 
> Cheers,
>    - Andreas
> 
> Jecel Assumpcao Jr wrote:
> > I'm posting this here since it is a policy issue, but I would be equally
> > happy to discuss this in vm-dev or squeak-dev if anyone thinks those
> > would be better options.
> > 
> > One interesting tool that we have in Squeak is the image version number.
> > This is supposed to allow a VM to check that it is compatible with some
> > image before trying to use it. Most images still use the original 6502
> > version, with the 64 bit images having a different number (68000, so it
> > seems we have a theme of "important microprocessors in Apple history").
> > 
> > I think that the policy so far has been to save this tool for something
> > really big, so changes to the VM in the past have not resulted in new
> > image version numbers. An example would be the introduction of
> > exceptions. Certainly the new VMs could fully handle old images - these
> > would just never trigger the newly added code. Unfortunately, old VMs
> > would not be able to deal with new images. They might seem to for quite
> > a while and only if an exception were actually triggered would things
> > break down. So casual tests might indicate bidirectional compatibility
> > even if specific tests would prove that not to be the case. So keeping
> > the old number would lead some people (but not most, who tend to use the
> > very latest VMs) into trouble while bumping up the number would make the
> > new VMs refuse to load images even though it is perfectly fine for them
> > to do so.
> > 
> > A simple solution would be for VMs to look for a list of compatible
> > image numbers rather than a single one. The new exception VMs would
> > accept either 6502 or 6503 images, for example. This would allow all
> > valid combinations to continue to work while preventing new images from
> > running on the old VMs where they could break. Of course, one worry is
> > that this list might grow to be a significant overhead for future VMs,
> > but that is unlikely to happen (specially if numbers are assigned wisely
> > so simple ranges could cover most cases).
> > 
> > It seems that the fruits of Eliot's Cog work is being adopted in pieces.
> > There are several VMs and images out there using his closure changes,
> > and I hope we will next see the Stack Pages, the Jitter, new header and
> > the cleaner bytecodes get widely used. This last upgrade will no doubt
> > force us to select a new image number, but will all the previous ones
> > stick with 6502?
> > 
> > An interesting thread on squeak-dev and vm-dev is about non ASCII input
> > from keyboard and clipboard. My own experience from this came late last
> > week when I was forced to reinstall all my applications after a hardware
> > replacement. So I went to download.squeak.org (nice to have this new
> > URL) and got the latest Linux VM for Squeak. This image (I use Celeste
> > for my email) seemed to work just fine as long as I was replying in
> > English, but pressing the key for ~ got me a debugger. Testing showed
> > that any deadkey character caused the same problem. A quick look at the
> > release history part of the Unix VM page showed that 3.10-1 had changed
> > the keyboard input. So I thought it was silly for me to try to use a 3.9
> > image with a 3.10-4 VM and tested a 3.10.2 image instead. But it has the
> > same error - I suppose only the Etoys image has the needed code to deal
> > with the new VM events. My opinion is that proper use of the image
> > version number could at least have generated a nice warning (no big deal
> > for me - I just got a 3.7-7 VM instead and am using that, but what about
> > "normal" users?).
> > 
> > I was hoping we could hold off all image changes for one big leap in
> > Squeak 5.0 since I have been comparing such a transition to the changes
> > Apple has gone through with the Mac and how disruptive this can be. But
> > now I feel this battle has been already lost and the current incremental
> > path will be the norm. So could we at least agree to use the only tool
> > we have, the image version number, to let people know what is supposed
> > to work and what isn't?
> > 
> > Many cheers,
> > -- Jecel