[squeak-dev] How does ImageSegmentLoader>>readObject work?

Eliot Miranda eliot.miranda at gmail.com
Wed May 23 18:25:43 UTC 2018

Hi Hannes,

On Wed, May 23, 2018 at 4:15 AM, H. Hirzel <hannes.hirzel at gmail.com> wrote:

> Hello
> this is a follow up mail on loading a project file (*.pr) saved in
> Squeak 3.10.2 [1] into Squeak 6.0a-#18000  [2]
> The code breaks at
>     ImageSegmentLoader>>readObject [3]
> I'd like to find out which class causes this problem.
> readObject
>         | header oop nWords class format |
>         header := self readUint32.
>         (header bitAnd: HeaderTypeMask) caseOf: {
>                 [HeaderTypeSizeAndClass] ->
>                         [nWords := header >> 2. class := self readUint32.
> header := self readUint32].
>                 [HeaderTypeClass] ->
> The header is a 32 bit integer denoting the class, I assume.
> I'd like to know which class causes the problem.
> How can I find out which class this 32 bit integer refers to?

There are three header formats, selected by a two bit field, the fourth
value identifying free objects, and hence never seen in an image segment.
The formats are

HeaderTypeSizeAndClass (0) The header is three words; the first is a size
field, the second is the class up, and the third is the header (containing
format, GC bits, etc)

HeaderTypeClass (1) The header is two words; the first is the class oop,
the second is the header (containing format, GC bits, etc, and in this case
a 6 bit word size field)

(HeaderTypeFree (2))

HeaderTypeShort (3): The header is one word, being the header including a 5
bit field (31 bitShift: 12) that is a zero-relative index into an array of
32 possible compact classes that in pre-Spur Squeaks are held in Smalltalk
compactClasses.  In ImageSegmentLoader they are in CompactClasses,
initialized in the class-side initialize method.  I would be willing to bet
real money that this is the kind of header that is failing, and that the
issue is missing compact classes such as MethodProperties.

Here's the layout of the header word in all three cases (taken from
ObjectMemory's class comment in the VMMaker/VMMaker.oscog package):

MSB 3 bits reserved for gc (mark, root, unused)
12 bits object hash (for HashSets) 5 bits compact class index 4 bits object
format 6 bits object size in 32-bit words LSB: 2 bits header type (0:
3-word, 1: 2-word, 2: forbidden, 3: 1-word)

Here's the initialization of CompactClasses from ImageSegmentLoader

CompactClasses := {CompiledMethod. nil. Array. nil.
LargePositiveInteger. Float. MethodDictionary. Association.
Point. Rectangle. ByteString. nil.
nil "was BlockContext; needs special handling". Context. nil. Bitmap.
nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil. nil}.

Looking at the V39 sources I can't see an initialization of the
compactClassesArray; recreateSpecialObjectsArray simply copies the existing
compactClassesArray.  Each image can have its own compact classes
(Behavior>>becomeCompact).  Therefore there has to be a copy of the compact
classes array saved in the image segment.  I can see traces of this but
don't know how the code works.  Look for compactClassesArray in the
3.9/3.10 sources and you may be able to find where the compact classes are
in an image segment.

Once you've located the compact classes you'll have to decide how to handle
the mapping.  The most important two things that have changed from 3.10 to
Spur are

1. the bytecode set and block implementation; 3.10 used Smalltalk-80 style
non-reentrant BlockContexts; as of Cog (4.0? 4.1?) we moved to reentrant
BlockClosure, where MethodContext (now Context) is used for block and
method activations.  There are 6 new byte codes used to implement closures;
see the class comment of EncoderForV3PlusClosures

2. 3.10 had an instance of MethodProperties as the penultimate literal in
all methods.  As of Cog (but this has nothing to do with the VM) methods
either have a selector in the penultimate literal, or an instance of
AdditionalMethodState (if the method has a pragma or properties), hence
saving considerable space.

To map from BlockContext to BlockClosure you'll have to revive the
decompiler for pre-closure bytecodes, decompile the block's method,
recompile to closure byte codes, and map PCs and temp vars appropriately.
This isn't easy in general, but it is possible, at least in theory, and
simple blocks may be simple (e.g. blocks that only take arguments).  (Think
about the problem as you would in understanding how the Debugger maps from
a context's pc and temporary variables to its display of the currently
executing expression, and the temporary variables and back.  At least in
4.0 and/or 4.1 we had the pc/variable mapping system working for both, and
we still have remnants of this; see DebuggerMethodMap's two subclasses,
DebuggerMethodMapForBlueBookMethods &
DebuggerMethodMapForClosureCompiledMethods.  You should be able to use
these to perform the mapping.

To map from MethodProperties to AdditionalMethodState in a CompiledMethod
is essentially trivial.  See if the MethodProperties has only a selector
and then simply use the selector in place of the MethodProperties.  If
there are additional properties, create an equivalent
AdditionalMethodState, copying across the properties, and use that in place
of the MethodProperties.

This is low-level enough that you may want to pair or at least consult with
myself or Bert.  I was the one who wrought all this change.  I apologise
for the severe inconvenience, but the current relative performance between
Cog and the 3.10 interpreter in part depends on these changes.


> Regards
> Hannes
> ------------------------------------------------------------
> --------------------------------------------------------------------
> [1] The project file has a workspace in it in addition to some
> RectangleMorphs and SimpleButtonMorphs which were loaded succesfully
> in another test.
> http://forum.world.st/The-Trunk-Morphic-kfr-1435-mcz-
> tp5077266p5077476.html
> The reference has a test project *.pr file attached.
> [2]
> http://lists.squeakfoundation.org/pipermail/squeak-dev/2018-
> May/198848.html
> New method added
> SmartRefStream  >> in the conversion protocol
> methodPropertiespps0
>        ^ AdditionalMethodState
> [3] readObject
>         | header oop nWords class format |
>         header := self readUint32.
>         (header bitAnd: HeaderTypeMask) caseOf: {
>                 [HeaderTypeSizeAndClass] ->
>                         [nWords := header >> 2. class := self readUint32.
> header := self readUint32].
>                 [HeaderTypeClass] ->
>                         [class := header - 1. header := self readUint32.
> nWords := header
> >> 2 bitAnd: 63].
>                 [HeaderTypeShort] ->
>                         [nWords := header >> 2 bitAnd: 63. class := header
> >> 12 bitAnd: 31].
>         } otherwise: [self error: 'unexpected free chunk'].
>         nWords := nWords - 1.   "nWords includes 1 header word"
>         oop := position.
>         ^[oopMap at: oop ifAbsentPut:
>                 [format := header >> 8 bitAnd: 15.
>                 "hash := header >> 17 bitAnd: 4095."
>                 self allocateObject: format class: class size: nWords]]
>                         ensure: [position := oop + (nWords * 4)]

best, Eliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20180523/bb83f3c4/attachment.html>

More information about the Squeak-dev mailing list