[Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

Chris Cunnington brasspen at gmail.com
Thu Aug 20 01:18:49 UTC 2015


As I've never seen (even on the wiki) an even halfway decent 
demonstration of using an ImageSegment, I'm providing on here.

     hex := Browser allInstances first.

     "let's check what we're saving to compare with later"
     (hex buildWith: ToolBuilder default) openInWorld

     exporting := (ImageSegment new copyFromRootsForExport: (Array with: 
hex)).

     “let’s put it on disk”
     exporting  writeForExport: 'browser.extSeg'.

     "Quit your image without saving. Actually, to get the full effect 
go to http://ftp.squeak.org and get a fresh image of the same kind you 
exported with. Drag browser.extSeg into the new image directory"

     “let’s pull it in from the disk”
     importing := (FileDirectory default readOnlyFileNamed: 
'browser.extSeg') fileInObjectAndCode.

     "let's check what we imported to see if it's what we saved"
     ((importing originalRoots first) buildWith: ToolBuilder default) 
openInWorld

Chris


On 2015-08-19 8:12 PM, David T. Lewis wrote:
> I think that image segments are a worthwhile idea, regardless of whether
> we are worried about support project saving and other image conversion
> issues. Here is a post by Dan Ingalls from 1999 that summarizes the work
> on image segments at that time:
>
>    http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>
> Dan also mentioned it in "the future of Squeak, 1999":
>
>    http://wiki.squeak.org/squeak/393
>
> It is quite clear that saving projects was a proof of concept to illustrate
> what might be done with image segments, but the overall motivation had more
> to do with exploring modularity and mechanisms for delivering minimal images.
>
> Tim Rowledge added a comment to that effect here:
>
>    http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>
> I will also note that Spur is not the first time we have needed to think
> about making image segments work on a new image format. When Dan Ingalls and
> Ian Piumarta announced the first 64-bit Squeak image, they said "We may ask for
> help from the Squeak community in converting the remaining plugins, and also
> image segment code."
>
>    http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-August/081383.html
>
> Quite a few of the things that Dan and Ian asked for help on have since been
> done, but I don't think that image segment support for the 64-bit image was
> among them. With the arrival of Spur, and the upcoming Spur 64-bit image, it
> would be great if we can put some thought and effort into doing this right.
>
> $0.02,
>
> Dave
>
>
> On Wed, Aug 19, 2015 at 02:09:32PM -0700, Eliot Miranda wrote:
>>   
>> Hi All, but especially Tobias,
>>
>>      as we know, ImageSegments are broken in Spur. They work occasionally,
>> but when stressed the VM ends up crashing.  The fundamental issue is that
>> the ImageSegment code makes assumptions about object ordering that Spur
>> violates.  For example, here's ImageSegment's install:
>>
>> install
>> "This operation retrieves the segment if necessary from file storage,
>> installs it in memory, and replaces (using become:) all the root stubs with
>> the reconstructed roots of the segment."
>>
>> | newRoots |
>> state = #onFile ifTrue: [self readFromFile].
>> state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
>> endMarker := segment nextObject. "for enumeration of objects"
>> endMarker == 0 ifTrue: [endMarker := 'End' clone]].
>> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>> state = #imported
>> ifTrue: ["just came in from exported file"
>> arrayOfRoots := newRoots]
>> ifFalse: [
>> arrayOfRoots elementsForwardIdentityTo: newRoots].
>> state := #inactive.
>> Beeper beepPrimitive
>>
>> So before the image segment bytes (the segment inst var) is loaded, the
>> object after it is assigned to endMarker, and if there isn't an object
>> after segment, a new object ('End' clone) is assigned to endMarker.
>>
>> This makes the assumption that objects are allocated in a strict order, and
>> therefore endMarker will always be the object after segment.
>>
>> Loading the segment via "newRoots := self loadSegmentFrom: segment
>> outPointers: outPointers" then turns segment into a zero-length WordArray
>> and its contents into the objects loaded by the segment.  Therefore, in the
>> V3 system, the objects loaded from segment can be enumerated starting at
>> segment nextObject and repeating until endMarker is found:
>>
>> allObjectsDo: aBlock
>> "Enumerate all objects that came from this segment.  NOTE this assumes that
>> the segment was created (and extracted).  After the segment has been
>> installed (install), this method allows you to enumerate its objects."
>> | obj |
>>
>> endMarker == nil ifTrue: [
>> ^ self error: 'Just extract and install, don''t writeToFile:'].
>> segment size ~= 1 ifTrue: [
>> ^ self error: 'Vestigial segment size must be 1 (version word)'].
>>
>> obj := segment nextObject.  "Start with the next object after the vestigial
>> header"
>> [obj == endMarker] whileFalse:  "Stop at the next object after the full
>> segment"
>> [aBlock value: obj.
>> obj := obj nextObject].  "Step through the objects installed from the
>> segment."
>>
>> Now, as written, this just won't work in Spur.
>>
>> a) the only place where there is any kind of stable order to objects is in
>> oldSpace, so segment /has/ to be forced to old space to have any chance of
>> its objects being in order when it gets converted from bytes to objects.
>>
>> b) 'End' clone will be in newSpace and so endMarker isn't reliable unless
>> it was obtained by segment nextObject when segment was already in oldSpace.
>>
>> So it is perhaps possible to fix ImageSegments in Spur by forcing segment
>> to oldSpace and being more careful with endMarker.  But I think there is a
>> better way.
>>
>> If the set of objects the segment contains can be obtained some how then
>> this set can be simply enumerated, not depending on nextObject.  The
>> primitive has to answer the array of roots, so its result can't be changed
>> to be the entire array.  But segment could be becomed into an Array of all
>> the objects in segment prior to it being loaded, in which case the above
>> would become
>>
>>
>> install
>> "This operation retrieves the segment if necessary from file storage,
>> installs it in memory, and replaces (using become:) all the root stubs with
>> the reconstructed roots of the segment."
>>
>> | newRoots |
>> state = #onFile ifTrue: [self readFromFile].
>> state = #onFileWithSymbols ifTrue:
>> [self readFromFileWithSymbols].
>> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>> state = #imported
>> ifTrue: "just came in from exported file"
>> [arrayOfRoots := newRoots]
>> ifFalse:
>> [arrayOfRoots elementsForwardIdentityTo: newRoots].
>> state := #inactive.
>> Beeper beepPrimitive
>>
>> allObjectsDo: aBlock
>> "Enumerate all objects that came from this segment.  NOTE this assumes that
>> the segment was created (and extracted).  After the segment has been
>> installed (install), this method allows you to enumerate its objects."
>> | obj |
>>
>> segment isArray ifFalse:
>> [^ self error: 'Segment hasn''t been loaded?'].
>>
>> segment do: aBlock
>>
>> and the endMarker instVar would be deleted.
>>
>> I am willing and ready to modify the primitive to convert the segment
>> correctly.  Who will volunteer to rewrite the image-level ImageSegment code
>> to use the new primitive?
>>
>>
>>
>> On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <eliot.miranda at gmail.com>
>> wrote:
>>
>>> Hi Tobias,
>>>
>>> On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <Das.Linux at gmx.de> wrote:
>>>
>>>> Hi all
>>>> On 13.08.2015, at 02:15, Eliot Miranda <eliot.miranda at gmail.com> wrote:
>>>>
>>>>> Hi Tobias,
>>>>>
>>>>>     forget that. I found it.  THis right?
>>>>>
>>>>> Trunk test suite for Spur
>>>>> Using existing cogspur r.3410
>>>>> cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410
>>>> /tmp/d20150812-28620-etbikj
>>>>>    image test suite
>>>>> VM: /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
>>>>> /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
>>>> -version
>>>>> /var/lib/jenkins/workspace/Trunk/default/tests.st
>>>>> spawning command 0 with timeout 1800 seconds:
>>>> "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak"
>>>> "-vm-sound-null" "-vm-display-null"
>>>> "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image"
>>>> "../tests.st"
>>>>> (Command started with PID 28643)
>>>>> 2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from
>>>> /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
>>>>> 2015-08-12T23:31:48.388+01:00: Running tests...
>>>>> setsockopt: Protocol not available
>>>>> setsockopt: Protocol not available
>>>>> 28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
>>>> protocol:s23_clnt.c:612:
>>>>> Recursive not understood error encountered
>>>>>
>>>>>
>>>>> I bet this is ImageSegment related.
>>>> It sure is.
>>>>
>>>>          BitmapStreamTests>testMatrixTransform2x3WithImageSegment
>>>> sends
>>>>          BitmapStreamTests>validateImageSegment
>>>>
>>>   Basically the ImageSegment code has been working in Spur on a hope and a
>>> prayer.  The code assumes objects are allocated in chronological order and
>>> that this order is preserved, not so with Spur.  So post image segment
>>> loading someObject/nextObject is used to enumerate the objects loaded.
>>> This can't work reliably in Spur.  I *think* (ok I hope) that I've
>>> implemented the segment load primitive in Spur to answer an Array of the
>>> objects loaded, so that these can be explicitly enumerated.
>>>
>>> So the job is a) to check that I have indeed implemented the primitive to
>>> do this and b) to rewrite the image segment loading code in the light of
>>> this.
>>>
>>> David, this is an example of something that isn't back portable and should
>>> not be back-ported.
>>>
>>> The most strange thing is that I cannot reproduce this on my Mac???
>>>> There the test does not crash the image???
>>>>
>>> Well then it may be a signed/unsigned bug in image loading instead.  On
>>> linux the image typically gets loaded quite high in the address space and
>>> so a good portion of the heap ends up above 0x7FFFFFFF, or negative
>>> territory if misinterpreted as signed integers.  On Mac and Windows the
>>> image tends to get loaded quite low and so has to be pretty big to fall
>>> foul of signedness issues.
>>>
>>>
>>> Busy right now but will check tomorrow.
>>>
>>>
>>>> Best regards
>>>>          -Tobias
>>>>
>>>>
>>>>> On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <eliot.miranda at gmail.com>
>>>> wrote:
>>>>> Hi Tobias,
>>>>>
>>>>> On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <Das.Linux at gmx.de> wrote:
>>>>>
>>>>> On 12.08.2015, at 20:55, Eliot Miranda <eliot.miranda at gmail.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>>
>>>>>>      Fabio's kindly done most of the changes.  But some questions
>>>> remain for more general discussion.  See below.
>>>>>> Fabio,  thanks so much for doing this, and so quickly!
>>>>>>
>>>>>> On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <
>>>> eliot.miranda at gmail.com> wrote:
>>>>>> Hi All,
>>>>>>
>>>>>>      who will update http://squeak.org/downloads/ to include the 5.0
>>>> release?
>>>>>> a) I suggest that the 5.0 all-in-one have a line in the left-hand
>>>> table that includes the 4.6 release and that it precede the 4.6 release in
>>>> the list.
>>>>>> Done.
>>>>>>
>>>>>>
>>>>>> b) the Trunk link points to TrunkImage.zip which contains a non-Spur
>>>> image.
>>>>>> So this is the biggie.  What should we do?
>>>>>>
>>>>>> - add a Trunk 5.0 build with a different link?  (Noooooo)
>>>>>>
>>>>>> - change  http://build.squeak.org/job/SqueakTrunk to build what it
>>>> says (Yessss please, who can do this?)
>>>>> Done: build.squeak.org/job/Trunk/
>>>>> But as I said several times, Spur/trunk test just crash for month???
>>>>>
>>>>> can you point me to the crash?  I'm looking at
>>>> http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink
>>>> and see 8 test failures but no crash.
>>>>>
>>>>>> - add a Squeak 4.6 build job?  Is that even worth it any more
>>>> considering 4.6 is released and stable?  If it is then what should it
>>>> build?  David?
>>>>>>
>>>>>> c) Spur VMs need to be linked to by an added line in the Virtual
>>>> Machines list.
>>>>>> Not applicable.  the VM links point to the root of my site so they
>>>> effectively point to both old and Spur VMs.
>>>>>>
>>>>>> d) Spur imagers need to be included in the Image and Changes list
>>>> under Custom Installation
>>>>>> Done.
>>>>>>
>>>>>>
>>>>>>
>>>>>> e) The SqueakV50.sources file also needs to be included in the
>>>> Current Sources list.
>>>>>> Done.
>>>>>>
>>>>>>
>>>>>> f) we could start a History list for the 5.0 line
>>>>>>
>>>>>> This is probably quite a big reorg on the page and not urgent.
>>>>>>
>>>>>>
>>>>>> So there are a few things to fix before 5.0 is freely downloadable.
>>>>>>
>>>>>> On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <ma.chris.m at gmail.com>
>>>> wrote:
>>>>>> In the 17 months since Squeak 4.5 was released, a huge development
>>>>>> effort took place to create the next generation virtual-machine for
>>>>>> the Squeak / Pharo / Newspeak family of programming systems.  Squeak
>>>>>> is the modern incarnation of the Smalltalk-80 programming environment
>>>>>> originally developed at the Xerox PARC.
>>>>>>
>>>>>> "Squeak 5" introduces this new VM and associated new memory model,
>>>>>> collectively referred to as "Spur".  Presented [1] by Eliot Miranda
>>>>>> and Cl??ment B??ra at the 2015 International Symposium on Memory
>>>>>> Management, this new VM affords Squeak applications a significant
>>>>>> boost in performance and memory management.  Among other
>>>>>> optimizations, the #become operation no longer requires a memory scan.
>>>>>> Object pinning and ephemerons are also now supported.  The release
>>>>>> notes [2] provide more details.
>>>>>>
>>>>>> The new memory model requires a new image file format.  Although this
>>>>>> new format results in about a 15% increased memory requirement for the
>>>>>> same number of 4.x objects, a new segmented heap allows memory to be
>>>>>> given back to the OS when its no longer needed, a great benefit for
>>>>>> application servers.
>>>>>>
>>>>>> As forward compatibility is as important to the Squeak community as
>>>>>> backward compatibility, Squeak 5 is delivers an image with identical
>>>>>> content as the recent 4.6 release.  Although this new Squeak 5 VM
>>>>>> cannot open images saved under the prior 4.x Cog format, objects and
>>>>>> code can be easily exported from the 4.x image and then imported into
>>>>>> Squeak 5.  Applications whose code runs strictly above the Smalltalk
>>>>>> meta layer will prove remarkably compatible with the new format, most
>>>>>> applications will require no changes whatsotever.
>>>>>>
>>>>>> Squeak 5 is the result of monumental effort by a tiny group of very
>>>>>> talented people, but its also just the beginning of yet a new effort;
>>>>>> Spur is just a stepping stone to a more ambitious goals planned over
>>>>>> the next five years.
>>>>>>
>>>>>> [1] -- A Partial Read Barrier for Efficient Support of Live
>>>>>> Object-oriented Programming
>>>>>>
>>>> http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
>>>>>> [2] -- Squeak 5 Release Notes
>>>>>> http://wiki.squeak.org/squeak/6207
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> _,,,^..^,,,_
>>> best, Eliot
>>>
>>
>>
>> -- 
>> _,,,^..^,,,_
>> best, Eliot
>



More information about the Squeak-dev mailing list