[Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN] Squeak 5]

Chris Cunnington brasspen at gmail.com
Thu Aug 20 01:33:37 UTC 2015


Sweet.

Chris

On 2015-08-19 9:31 PM, Mariano Martinez Peck wrote:
> Hi guys,
>
> As part of my PhD thesis I did take a deep look to ImageSegment before 
> starting with Fuel. I wrote a journal paper about my experiments with 
> ImageSegment which I thinks provides quite a documentation that is not 
> written anywhere. Hope it helps: 
> http://rmod.lille.inria.fr/archives/papers/Mart11c-COMLAN-ObjectSwapping.pdf
>
> Best,
>
>
>
> On Wed, Aug 19, 2015 at 10:18 PM, Chris Cunnington <brasspen at gmail.com 
> <mailto:brasspen at gmail.com>> wrote:
>
>     As I've never seen (even on the wiki) an even halfway decent
>     demonstration of using an ImageSegment, I'm providing on here.
>
>         hex := Browser allInstances first.
>
>         "let's check what we're saving to compare with later"
>         (hex buildWith: ToolBuilder default) openInWorld
>
>         exporting := (ImageSegment new copyFromRootsForExport: (Array
>     with: hex)).
>
>         “let’s put it on disk”
>         exporting  writeForExport: 'browser.extSeg'.
>
>         "Quit your image without saving. Actually, to get the full
>     effect go to http://ftp.squeak.org and get a fresh image of the
>     same kind you exported with. Drag browser.extSeg into the new
>     image directory"
>
>         “let’s pull it in from the disk”
>         importing := (FileDirectory default readOnlyFileNamed:
>     'browser.extSeg') fileInObjectAndCode.
>
>         "let's check what we imported to see if it's what we saved"
>         ((importing originalRoots first) buildWith: ToolBuilder
>     default) openInWorld
>
>     Chris
>
>
>
>     On 2015-08-19 8:12 PM, David T. Lewis wrote:
>
>         I think that image segments are a worthwhile idea, regardless
>         of whether
>         we are worried about support project saving and other image
>         conversion
>         issues. Here is a post by Dan Ingalls from 1999 that
>         summarizes the work
>         on image segments at that time:
>
>         http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>
>         Dan also mentioned it in "the future of Squeak, 1999":
>
>         http://wiki.squeak.org/squeak/393
>
>         It is quite clear that saving projects was a proof of concept
>         to illustrate
>         what might be done with image segments, but the overall
>         motivation had more
>         to do with exploring modularity and mechanisms for delivering
>         minimal images.
>
>         Tim Rowledge added a comment to that effect here:
>
>         http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>
>         I will also note that Spur is not the first time we have
>         needed to think
>         about making image segments work on a new image format. When
>         Dan Ingalls and
>         Ian Piumarta announced the first 64-bit Squeak image, they
>         said "We may ask for
>         help from the Squeak community in converting the remaining
>         plugins, and also
>         image segment code."
>
>         http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-August/081383.html
>
>         Quite a few of the things that Dan and Ian asked for help on
>         have since been
>         done, but I don't think that image segment support for the
>         64-bit image was
>         among them. With the arrival of Spur, and the upcoming Spur
>         64-bit image, it
>         would be great if we can put some thought and effort into
>         doing this right.
>
>         $0.02,
>
>         Dave
>
>
>         On Wed, Aug 19, 2015 at 02:09:32PM -0700, Eliot Miranda wrote:
>
>               Hi All, but especially Tobias,
>
>                  as we know, ImageSegments are broken in Spur. They
>             work occasionally,
>             but when stressed the VM ends up crashing.  The
>             fundamental issue is that
>             the ImageSegment code makes assumptions about object
>             ordering that Spur
>             violates.  For example, here's ImageSegment's install:
>
>             install
>             "This operation retrieves the segment if necessary from
>             file storage,
>             installs it in memory, and replaces (using become:) all
>             the root stubs with
>             the reconstructed roots of the segment."
>
>             | newRoots |
>             state = #onFile ifTrue: [self readFromFile].
>             state = #onFileWithSymbols ifTrue: [self
>             readFromFileWithSymbols.
>             endMarker := segment nextObject. "for enumeration of objects"
>             endMarker == 0 ifTrue: [endMarker := 'End' clone]].
>             (state = #active) | (state = #imported) ifFalse: [self
>             errorWrongState].
>             newRoots := self loadSegmentFrom: segment outPointers:
>             outPointers.
>             state = #imported
>             ifTrue: ["just came in from exported file"
>             arrayOfRoots := newRoots]
>             ifFalse: [
>             arrayOfRoots elementsForwardIdentityTo: newRoots].
>             state := #inactive.
>             Beeper beepPrimitive
>
>             So before the image segment bytes (the segment inst var)
>             is loaded, the
>             object after it is assigned to endMarker, and if there
>             isn't an object
>             after segment, a new object ('End' clone) is assigned to
>             endMarker.
>
>             This makes the assumption that objects are allocated in a
>             strict order, and
>             therefore endMarker will always be the object after segment.
>
>             Loading the segment via "newRoots := self loadSegmentFrom:
>             segment
>             outPointers: outPointers" then turns segment into a
>             zero-length WordArray
>             and its contents into the objects loaded by the segment. 
>             Therefore, in the
>             V3 system, the objects loaded from segment can be
>             enumerated starting at
>             segment nextObject and repeating until endMarker is found:
>
>             allObjectsDo: aBlock
>             "Enumerate all objects that came from this segment. NOTE
>             this assumes that
>             the segment was created (and extracted).  After the
>             segment has been
>             installed (install), this method allows you to enumerate
>             its objects."
>             | obj |
>
>             endMarker == nil ifTrue: [
>             ^ self error: 'Just extract and install, don''t
>             writeToFile:'].
>             segment size ~= 1 ifTrue: [
>             ^ self error: 'Vestigial segment size must be 1 (version
>             word)'].
>
>             obj := segment nextObject.  "Start with the next object
>             after the vestigial
>             header"
>             [obj == endMarker] whileFalse:  "Stop at the next object
>             after the full
>             segment"
>             [aBlock value: obj.
>             obj := obj nextObject].  "Step through the objects
>             installed from the
>             segment."
>
>             Now, as written, this just won't work in Spur.
>
>             a) the only place where there is any kind of stable order
>             to objects is in
>             oldSpace, so segment /has/ to be forced to old space to
>             have any chance of
>             its objects being in order when it gets converted from
>             bytes to objects.
>
>             b) 'End' clone will be in newSpace and so endMarker isn't
>             reliable unless
>             it was obtained by segment nextObject when segment was
>             already in oldSpace.
>
>             So it is perhaps possible to fix ImageSegments in Spur by
>             forcing segment
>             to oldSpace and being more careful with endMarker. But I
>             think there is a
>             better way.
>
>             If the set of objects the segment contains can be obtained
>             some how then
>             this set can be simply enumerated, not depending on
>             nextObject.  The
>             primitive has to answer the array of roots, so its result
>             can't be changed
>             to be the entire array.  But segment could be becomed into
>             an Array of all
>             the objects in segment prior to it being loaded, in which
>             case the above
>             would become
>
>
>             install
>             "This operation retrieves the segment if necessary from
>             file storage,
>             installs it in memory, and replaces (using become:) all
>             the root stubs with
>             the reconstructed roots of the segment."
>
>             | newRoots |
>             state = #onFile ifTrue: [self readFromFile].
>             state = #onFileWithSymbols ifTrue:
>             [self readFromFileWithSymbols].
>             (state = #active) | (state = #imported) ifFalse: [self
>             errorWrongState].
>             newRoots := self loadSegmentFrom: segment outPointers:
>             outPointers.
>             state = #imported
>             ifTrue: "just came in from exported file"
>             [arrayOfRoots := newRoots]
>             ifFalse:
>             [arrayOfRoots elementsForwardIdentityTo: newRoots].
>             state := #inactive.
>             Beeper beepPrimitive
>
>             allObjectsDo: aBlock
>             "Enumerate all objects that came from this segment. NOTE
>             this assumes that
>             the segment was created (and extracted).  After the
>             segment has been
>             installed (install), this method allows you to enumerate
>             its objects."
>             | obj |
>
>             segment isArray ifFalse:
>             [^ self error: 'Segment hasn''t been loaded?'].
>
>             segment do: aBlock
>
>             and the endMarker instVar would be deleted.
>
>             I am willing and ready to modify the primitive to convert
>             the segment
>             correctly.  Who will volunteer to rewrite the image-level
>             ImageSegment code
>             to use the new primitive?
>
>
>
>             On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda
>             <eliot.miranda at gmail.com <mailto:eliot.miranda at gmail.com>>
>             wrote:
>
>                 Hi Tobias,
>
>                 On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape
>                 <Das.Linux at gmx.de <mailto:Das.Linux at gmx.de>> wrote:
>
>                     Hi all
>                     On 13.08.2015, at 02:15, Eliot Miranda
>                     <eliot.miranda at gmail.com
>                     <mailto:eliot.miranda at gmail.com>> wrote:
>
>                         Hi Tobias,
>
>                             forget that. I found it.  THis right?
>
>                         Trunk test suite for Spur
>                         Using existing cogspur r.3410
>                         cp -r
>                         /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410
>
>                     /tmp/d20150812-28620-etbikj
>
>                            image test suite
>                         VM:
>                         /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
>                         /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
>
>                     -version
>
>                         /var/lib/jenkins/workspace/Trunk/default/tests.st
>                         <http://tests.st>
>                         spawning command 0 with timeout 1800 seconds:
>
>                     "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak"
>                     "-vm-sound-null" "-vm-display-null"
>                     "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image"
>                     "../tests.st <http://tests.st>"
>
>                         (Command started with PID 28643)
>                         2015-08-12T23:31:48.17+01:00: Loading Hudson
>                         build tools... from
>
>                     /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
>
>                         2015-08-12T23:31:48.388+01:00: Running tests...
>                         setsockopt: Protocol not available
>                         setsockopt: Protocol not available
>                         28646:error:140770FC:SSL
>                         routines:SSL23_GET_SERVER_HELLO:unknown
>
>                     protocol:s23_clnt.c:612:
>
>                         Recursive not understood error encountered
>
>
>                         I bet this is ImageSegment related.
>
>                     It sure is.
>
>                      BitmapStreamTests>testMatrixTransform2x3WithImageSegment
>                     sends
>                      BitmapStreamTests>validateImageSegment
>
>                   Basically the ImageSegment code has been working in
>                 Spur on a hope and a
>                 prayer.  The code assumes objects are allocated in
>                 chronological order and
>                 that this order is preserved, not so with Spur. So
>                 post image segment
>                 loading someObject/nextObject is used to enumerate the
>                 objects loaded.
>                 This can't work reliably in Spur.  I *think* (ok I
>                 hope) that I've
>                 implemented the segment load primitive in Spur to
>                 answer an Array of the
>                 objects loaded, so that these can be explicitly
>                 enumerated.
>
>                 So the job is a) to check that I have indeed
>                 implemented the primitive to
>                 do this and b) to rewrite the image segment loading
>                 code in the light of
>                 this.
>
>                 David, this is an example of something that isn't back
>                 portable and should
>                 not be back-ported.
>
>                 The most strange thing is that I cannot reproduce this
>                 on my Mac???
>
>                     There the test does not crash the image???
>
>                 Well then it may be a signed/unsigned bug in image
>                 loading instead.  On
>                 linux the image typically gets loaded quite high in
>                 the address space and
>                 so a good portion of the heap ends up above
>                 0x7FFFFFFF, or negative
>                 territory if misinterpreted as signed integers. On Mac
>                 and Windows the
>                 image tends to get loaded quite low and so has to be
>                 pretty big to fall
>                 foul of signedness issues.
>
>
>                 Busy right now but will check tomorrow.
>
>
>                     Best regards
>                              -Tobias
>
>
>                         On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda
>                         <eliot.miranda at gmail.com
>                         <mailto:eliot.miranda at gmail.com>>
>
>                     wrote:
>
>                         Hi Tobias,
>
>                         On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape
>                         <Das.Linux at gmx.de <mailto:Das.Linux at gmx.de>>
>                         wrote:
>
>                         On 12.08.2015, at 20:55, Eliot Miranda
>                         <eliot.miranda at gmail.com
>                         <mailto:eliot.miranda at gmail.com>> wrote:
>
>                             Hi All,
>
>
>                                  Fabio's kindly done most of the
>                             changes.  But some questions
>
>                     remain for more general discussion.  See below.
>
>                             Fabio,  thanks so much for doing this, and
>                             so quickly!
>
>                             On Tue, Aug 11, 2015 at 8:51 PM, Eliot
>                             Miranda <
>
>                     eliot.miranda at gmail.com
>                     <mailto:eliot.miranda at gmail.com>> wrote:
>
>                             Hi All,
>
>                                  who will update
>                             http://squeak.org/downloads/ to include
>                             the 5.0
>
>                     release?
>
>                             a) I suggest that the 5.0 all-in-one have
>                             a line in the left-hand
>
>                     table that includes the 4.6 release and that it
>                     precede the 4.6 release in
>                     the list.
>
>                             Done.
>
>
>                             b) the Trunk link points to TrunkImage.zip
>                             which contains a non-Spur
>
>                     image.
>
>                             So this is the biggie.  What should we do?
>
>                             - add a Trunk 5.0 build with a different
>                             link?  (Noooooo)
>
>                             - change
>                             http://build.squeak.org/job/SqueakTrunk to
>                             build what it
>
>                     says (Yessss please, who can do this?)
>
>                         Done: build.squeak.org/job/Trunk/
>                         <http://build.squeak.org/job/Trunk/>
>                         But as I said several times, Spur/trunk test
>                         just crash for month???
>
>                         can you point me to the crash?  I'm looking at
>
>                     http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink
>                     and see 8 test failures but no crash.
>
>
>                             - add a Squeak 4.6 build job?  Is that
>                             even worth it any more
>
>                     considering 4.6 is released and stable?  If it is
>                     then what should it
>                     build?  David?
>
>
>                             c) Spur VMs need to be linked to by an
>                             added line in the Virtual
>
>                     Machines list.
>
>                             Not applicable.  the VM links point to the
>                             root of my site so they
>
>                     effectively point to both old and Spur VMs.
>
>
>                             d) Spur imagers need to be included in the
>                             Image and Changes list
>
>                     under Custom Installation
>
>                             Done.
>
>
>
>                             e) The SqueakV50.sources file also needs
>                             to be included in the
>
>                     Current Sources list.
>
>                             Done.
>
>
>                             f) we could start a History list for the
>                             5.0 line
>
>                             This is probably quite a big reorg on the
>                             page and not urgent.
>
>
>                             So there are a few things to fix before
>                             5.0 is freely downloadable.
>
>                             On Tue, Aug 11, 2015 at 8:23 PM, Chris
>                             Muller <ma.chris.m at gmail.com
>                             <mailto:ma.chris.m at gmail.com>>
>
>                     wrote:
>
>                             In the 17 months since Squeak 4.5 was
>                             released, a huge development
>                             effort took place to create the next
>                             generation virtual-machine for
>                             the Squeak / Pharo / Newspeak family of
>                             programming systems.  Squeak
>                             is the modern incarnation of the
>                             Smalltalk-80 programming environment
>                             originally developed at the Xerox PARC.
>
>                             "Squeak 5" introduces this new VM and
>                             associated new memory model,
>                             collectively referred to as "Spur".
>                             Presented [1] by Eliot Miranda
>                             and Cl??ment B??ra at the 2015
>                             International Symposium on Memory
>                             Management, this new VM affords Squeak
>                             applications a significant
>                             boost in performance and memory
>                             management. Among other
>                             optimizations, the #become operation no
>                             longer requires a memory scan.
>                             Object pinning and ephemerons are also now
>                             supported.  The release
>                             notes [2] provide more details.
>
>                             The new memory model requires a new image
>                             file format.  Although this
>                             new format results in about a 15%
>                             increased memory requirement for the
>                             same number of 4.x objects, a new
>                             segmented heap allows memory to be
>                             given back to the OS when its no longer
>                             needed, a great benefit for
>                             application servers.
>
>                             As forward compatibility is as important
>                             to the Squeak community as
>                             backward compatibility, Squeak 5 is
>                             delivers an image with identical
>                             content as the recent 4.6 release. 
>                             Although this new Squeak 5 VM
>                             cannot open images saved under the prior
>                             4.x Cog format, objects and
>                             code can be easily exported from the 4.x
>                             image and then imported into
>                             Squeak 5.  Applications whose code runs
>                             strictly above the Smalltalk
>                             meta layer will prove remarkably
>                             compatible with the new format, most
>                             applications will require no changes
>                             whatsotever.
>
>                             Squeak 5 is the result of monumental
>                             effort by a tiny group of very
>                             talented people, but its also just the
>                             beginning of yet a new effort;
>                             Spur is just a stepping stone to a more
>                             ambitious goals planned over
>                             the next five years.
>
>                             [1] -- A Partial Read Barrier for
>                             Efficient Support of Live
>                             Object-oriented Programming
>
>                     http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
>
>                             [2] -- Squeak 5 Release Notes
>                             http://wiki.squeak.org/squeak/6207
>
>
>
>
>
>
>                 --
>                 _,,,^..^,,,_
>                 best, Eliot
>
>
>
>             -- 
>             _,,,^..^,,,_
>             best, Eliot
>
>
>
>
>
>
>
> -- 
> Mariano
> http://marianopeck.wordpress.com
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20150819/41db2a7d/attachment.htm


More information about the Squeak-dev mailing list