[Vm-dev] Squeak 5 Image Segment Work [was [squeak-dev] [ANN]
Squeak 5]
Chris Cunnington
brasspen at gmail.com
Thu Aug 20 01:33:37 UTC 2015
Sweet.
Chris
On 2015-08-19 9:31 PM, Mariano Martinez Peck wrote:
> Hi guys,
>
> As part of my PhD thesis I did take a deep look to ImageSegment before
> starting with Fuel. I wrote a journal paper about my experiments with
> ImageSegment which I thinks provides quite a documentation that is not
> written anywhere. Hope it helps:
> http://rmod.lille.inria.fr/archives/papers/Mart11c-COMLAN-ObjectSwapping.pdf
>
> Best,
>
>
>
> On Wed, Aug 19, 2015 at 10:18 PM, Chris Cunnington <brasspen at gmail.com
> <mailto:brasspen at gmail.com>> wrote:
>
> As I've never seen (even on the wiki) an even halfway decent
> demonstration of using an ImageSegment, I'm providing on here.
>
> hex := Browser allInstances first.
>
> "let's check what we're saving to compare with later"
> (hex buildWith: ToolBuilder default) openInWorld
>
> exporting := (ImageSegment new copyFromRootsForExport: (Array
> with: hex)).
>
> “let’s put it on disk”
> exporting writeForExport: 'browser.extSeg'.
>
> "Quit your image without saving. Actually, to get the full
> effect go to http://ftp.squeak.org and get a fresh image of the
> same kind you exported with. Drag browser.extSeg into the new
> image directory"
>
> “let’s pull it in from the disk”
> importing := (FileDirectory default readOnlyFileNamed:
> 'browser.extSeg') fileInObjectAndCode.
>
> "let's check what we imported to see if it's what we saved"
> ((importing originalRoots first) buildWith: ToolBuilder
> default) openInWorld
>
> Chris
>
>
>
> On 2015-08-19 8:12 PM, David T. Lewis wrote:
>
> I think that image segments are a worthwhile idea, regardless
> of whether
> we are worried about support project saving and other image
> conversion
> issues. Here is a post by Dan Ingalls from 1999 that
> summarizes the work
> on image segments at that time:
>
> http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>
> Dan also mentioned it in "the future of Squeak, 1999":
>
> http://wiki.squeak.org/squeak/393
>
> It is quite clear that saving projects was a proof of concept
> to illustrate
> what might be done with image segments, but the overall
> motivation had more
> to do with exploring modularity and mechanisms for delivering
> minimal images.
>
> Tim Rowledge added a comment to that effect here:
>
> http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>
> I will also note that Spur is not the first time we have
> needed to think
> about making image segments work on a new image format. When
> Dan Ingalls and
> Ian Piumarta announced the first 64-bit Squeak image, they
> said "We may ask for
> help from the Squeak community in converting the remaining
> plugins, and also
> image segment code."
>
> http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-August/081383.html
>
> Quite a few of the things that Dan and Ian asked for help on
> have since been
> done, but I don't think that image segment support for the
> 64-bit image was
> among them. With the arrival of Spur, and the upcoming Spur
> 64-bit image, it
> would be great if we can put some thought and effort into
> doing this right.
>
> $0.02,
>
> Dave
>
>
> On Wed, Aug 19, 2015 at 02:09:32PM -0700, Eliot Miranda wrote:
>
> Hi All, but especially Tobias,
>
> as we know, ImageSegments are broken in Spur. They
> work occasionally,
> but when stressed the VM ends up crashing. The
> fundamental issue is that
> the ImageSegment code makes assumptions about object
> ordering that Spur
> violates. For example, here's ImageSegment's install:
>
> install
> "This operation retrieves the segment if necessary from
> file storage,
> installs it in memory, and replaces (using become:) all
> the root stubs with
> the reconstructed roots of the segment."
>
> | newRoots |
> state = #onFile ifTrue: [self readFromFile].
> state = #onFileWithSymbols ifTrue: [self
> readFromFileWithSymbols.
> endMarker := segment nextObject. "for enumeration of objects"
> endMarker == 0 ifTrue: [endMarker := 'End' clone]].
> (state = #active) | (state = #imported) ifFalse: [self
> errorWrongState].
> newRoots := self loadSegmentFrom: segment outPointers:
> outPointers.
> state = #imported
> ifTrue: ["just came in from exported file"
> arrayOfRoots := newRoots]
> ifFalse: [
> arrayOfRoots elementsForwardIdentityTo: newRoots].
> state := #inactive.
> Beeper beepPrimitive
>
> So before the image segment bytes (the segment inst var)
> is loaded, the
> object after it is assigned to endMarker, and if there
> isn't an object
> after segment, a new object ('End' clone) is assigned to
> endMarker.
>
> This makes the assumption that objects are allocated in a
> strict order, and
> therefore endMarker will always be the object after segment.
>
> Loading the segment via "newRoots := self loadSegmentFrom:
> segment
> outPointers: outPointers" then turns segment into a
> zero-length WordArray
> and its contents into the objects loaded by the segment.
> Therefore, in the
> V3 system, the objects loaded from segment can be
> enumerated starting at
> segment nextObject and repeating until endMarker is found:
>
> allObjectsDo: aBlock
> "Enumerate all objects that came from this segment. NOTE
> this assumes that
> the segment was created (and extracted). After the
> segment has been
> installed (install), this method allows you to enumerate
> its objects."
> | obj |
>
> endMarker == nil ifTrue: [
> ^ self error: 'Just extract and install, don''t
> writeToFile:'].
> segment size ~= 1 ifTrue: [
> ^ self error: 'Vestigial segment size must be 1 (version
> word)'].
>
> obj := segment nextObject. "Start with the next object
> after the vestigial
> header"
> [obj == endMarker] whileFalse: "Stop at the next object
> after the full
> segment"
> [aBlock value: obj.
> obj := obj nextObject]. "Step through the objects
> installed from the
> segment."
>
> Now, as written, this just won't work in Spur.
>
> a) the only place where there is any kind of stable order
> to objects is in
> oldSpace, so segment /has/ to be forced to old space to
> have any chance of
> its objects being in order when it gets converted from
> bytes to objects.
>
> b) 'End' clone will be in newSpace and so endMarker isn't
> reliable unless
> it was obtained by segment nextObject when segment was
> already in oldSpace.
>
> So it is perhaps possible to fix ImageSegments in Spur by
> forcing segment
> to oldSpace and being more careful with endMarker. But I
> think there is a
> better way.
>
> If the set of objects the segment contains can be obtained
> some how then
> this set can be simply enumerated, not depending on
> nextObject. The
> primitive has to answer the array of roots, so its result
> can't be changed
> to be the entire array. But segment could be becomed into
> an Array of all
> the objects in segment prior to it being loaded, in which
> case the above
> would become
>
>
> install
> "This operation retrieves the segment if necessary from
> file storage,
> installs it in memory, and replaces (using become:) all
> the root stubs with
> the reconstructed roots of the segment."
>
> | newRoots |
> state = #onFile ifTrue: [self readFromFile].
> state = #onFileWithSymbols ifTrue:
> [self readFromFileWithSymbols].
> (state = #active) | (state = #imported) ifFalse: [self
> errorWrongState].
> newRoots := self loadSegmentFrom: segment outPointers:
> outPointers.
> state = #imported
> ifTrue: "just came in from exported file"
> [arrayOfRoots := newRoots]
> ifFalse:
> [arrayOfRoots elementsForwardIdentityTo: newRoots].
> state := #inactive.
> Beeper beepPrimitive
>
> allObjectsDo: aBlock
> "Enumerate all objects that came from this segment. NOTE
> this assumes that
> the segment was created (and extracted). After the
> segment has been
> installed (install), this method allows you to enumerate
> its objects."
> | obj |
>
> segment isArray ifFalse:
> [^ self error: 'Segment hasn''t been loaded?'].
>
> segment do: aBlock
>
> and the endMarker instVar would be deleted.
>
> I am willing and ready to modify the primitive to convert
> the segment
> correctly. Who will volunteer to rewrite the image-level
> ImageSegment code
> to use the new primitive?
>
>
>
> On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda
> <eliot.miranda at gmail.com <mailto:eliot.miranda at gmail.com>>
> wrote:
>
> Hi Tobias,
>
> On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape
> <Das.Linux at gmx.de <mailto:Das.Linux at gmx.de>> wrote:
>
> Hi all
> On 13.08.2015, at 02:15, Eliot Miranda
> <eliot.miranda at gmail.com
> <mailto:eliot.miranda at gmail.com>> wrote:
>
> Hi Tobias,
>
> forget that. I found it. THis right?
>
> Trunk test suite for Spur
> Using existing cogspur r.3410
> cp -r
> /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410
>
> /tmp/d20150812-28620-etbikj
>
> image test suite
> VM:
> /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
> /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
>
> -version
>
> /var/lib/jenkins/workspace/Trunk/default/tests.st
> <http://tests.st>
> spawning command 0 with timeout 1800 seconds:
>
> "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak"
> "-vm-sound-null" "-vm-display-null"
> "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image"
> "../tests.st <http://tests.st>"
>
> (Command started with PID 28643)
> 2015-08-12T23:31:48.17+01:00: Loading Hudson
> build tools... from
>
> /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
>
> 2015-08-12T23:31:48.388+01:00: Running tests...
> setsockopt: Protocol not available
> setsockopt: Protocol not available
> 28646:error:140770FC:SSL
> routines:SSL23_GET_SERVER_HELLO:unknown
>
> protocol:s23_clnt.c:612:
>
> Recursive not understood error encountered
>
>
> I bet this is ImageSegment related.
>
> It sure is.
>
> BitmapStreamTests>testMatrixTransform2x3WithImageSegment
> sends
> BitmapStreamTests>validateImageSegment
>
> Basically the ImageSegment code has been working in
> Spur on a hope and a
> prayer. The code assumes objects are allocated in
> chronological order and
> that this order is preserved, not so with Spur. So
> post image segment
> loading someObject/nextObject is used to enumerate the
> objects loaded.
> This can't work reliably in Spur. I *think* (ok I
> hope) that I've
> implemented the segment load primitive in Spur to
> answer an Array of the
> objects loaded, so that these can be explicitly
> enumerated.
>
> So the job is a) to check that I have indeed
> implemented the primitive to
> do this and b) to rewrite the image segment loading
> code in the light of
> this.
>
> David, this is an example of something that isn't back
> portable and should
> not be back-ported.
>
> The most strange thing is that I cannot reproduce this
> on my Mac???
>
> There the test does not crash the image???
>
> Well then it may be a signed/unsigned bug in image
> loading instead. On
> linux the image typically gets loaded quite high in
> the address space and
> so a good portion of the heap ends up above
> 0x7FFFFFFF, or negative
> territory if misinterpreted as signed integers. On Mac
> and Windows the
> image tends to get loaded quite low and so has to be
> pretty big to fall
> foul of signedness issues.
>
>
> Busy right now but will check tomorrow.
>
>
> Best regards
> -Tobias
>
>
> On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda
> <eliot.miranda at gmail.com
> <mailto:eliot.miranda at gmail.com>>
>
> wrote:
>
> Hi Tobias,
>
> On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape
> <Das.Linux at gmx.de <mailto:Das.Linux at gmx.de>>
> wrote:
>
> On 12.08.2015, at 20:55, Eliot Miranda
> <eliot.miranda at gmail.com
> <mailto:eliot.miranda at gmail.com>> wrote:
>
> Hi All,
>
>
> Fabio's kindly done most of the
> changes. But some questions
>
> remain for more general discussion. See below.
>
> Fabio, thanks so much for doing this, and
> so quickly!
>
> On Tue, Aug 11, 2015 at 8:51 PM, Eliot
> Miranda <
>
> eliot.miranda at gmail.com
> <mailto:eliot.miranda at gmail.com>> wrote:
>
> Hi All,
>
> who will update
> http://squeak.org/downloads/ to include
> the 5.0
>
> release?
>
> a) I suggest that the 5.0 all-in-one have
> a line in the left-hand
>
> table that includes the 4.6 release and that it
> precede the 4.6 release in
> the list.
>
> Done.
>
>
> b) the Trunk link points to TrunkImage.zip
> which contains a non-Spur
>
> image.
>
> So this is the biggie. What should we do?
>
> - add a Trunk 5.0 build with a different
> link? (Noooooo)
>
> - change
> http://build.squeak.org/job/SqueakTrunk to
> build what it
>
> says (Yessss please, who can do this?)
>
> Done: build.squeak.org/job/Trunk/
> <http://build.squeak.org/job/Trunk/>
> But as I said several times, Spur/trunk test
> just crash for month???
>
> can you point me to the crash? I'm looking at
>
> http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink
> and see 8 test failures but no crash.
>
>
> - add a Squeak 4.6 build job? Is that
> even worth it any more
>
> considering 4.6 is released and stable? If it is
> then what should it
> build? David?
>
>
> c) Spur VMs need to be linked to by an
> added line in the Virtual
>
> Machines list.
>
> Not applicable. the VM links point to the
> root of my site so they
>
> effectively point to both old and Spur VMs.
>
>
> d) Spur imagers need to be included in the
> Image and Changes list
>
> under Custom Installation
>
> Done.
>
>
>
> e) The SqueakV50.sources file also needs
> to be included in the
>
> Current Sources list.
>
> Done.
>
>
> f) we could start a History list for the
> 5.0 line
>
> This is probably quite a big reorg on the
> page and not urgent.
>
>
> So there are a few things to fix before
> 5.0 is freely downloadable.
>
> On Tue, Aug 11, 2015 at 8:23 PM, Chris
> Muller <ma.chris.m at gmail.com
> <mailto:ma.chris.m at gmail.com>>
>
> wrote:
>
> In the 17 months since Squeak 4.5 was
> released, a huge development
> effort took place to create the next
> generation virtual-machine for
> the Squeak / Pharo / Newspeak family of
> programming systems. Squeak
> is the modern incarnation of the
> Smalltalk-80 programming environment
> originally developed at the Xerox PARC.
>
> "Squeak 5" introduces this new VM and
> associated new memory model,
> collectively referred to as "Spur".
> Presented [1] by Eliot Miranda
> and Cl??ment B??ra at the 2015
> International Symposium on Memory
> Management, this new VM affords Squeak
> applications a significant
> boost in performance and memory
> management. Among other
> optimizations, the #become operation no
> longer requires a memory scan.
> Object pinning and ephemerons are also now
> supported. The release
> notes [2] provide more details.
>
> The new memory model requires a new image
> file format. Although this
> new format results in about a 15%
> increased memory requirement for the
> same number of 4.x objects, a new
> segmented heap allows memory to be
> given back to the OS when its no longer
> needed, a great benefit for
> application servers.
>
> As forward compatibility is as important
> to the Squeak community as
> backward compatibility, Squeak 5 is
> delivers an image with identical
> content as the recent 4.6 release.
> Although this new Squeak 5 VM
> cannot open images saved under the prior
> 4.x Cog format, objects and
> code can be easily exported from the 4.x
> image and then imported into
> Squeak 5. Applications whose code runs
> strictly above the Smalltalk
> meta layer will prove remarkably
> compatible with the new format, most
> applications will require no changes
> whatsotever.
>
> Squeak 5 is the result of monumental
> effort by a tiny group of very
> talented people, but its also just the
> beginning of yet a new effort;
> Spur is just a stepping stone to a more
> ambitious goals planned over
> the next five years.
>
> [1] -- A Partial Read Barrier for
> Efficient Support of Live
> Object-oriented Programming
>
> http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
>
> [2] -- Squeak 5 Release Notes
> http://wiki.squeak.org/squeak/6207
>
>
>
>
>
>
> --
> _,,,^..^,,,_
> best, Eliot
>
>
>
> --
> _,,,^..^,,,_
> best, Eliot
>
>
>
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20150819/41db2a7d/attachment.htm
More information about the Squeak-dev
mailing list
|