Hi guys,

As part of my PhD thesis I did take a deep look to ImageSegment before
starting with Fuel. I wrote a journal paper about my experiments with
ImageSegment which I thinks provides quite a documentation that is not
written anywhere. Hope it helps:
http://rmod.lille.inria.fr/archives/papers/Mart11c-COMLAN-ObjectSwapping.pdf

Best,



On Wed, Aug 19, 2015 at 10:18 PM, Chris Cunnington <brasspen at gmail.com>
wrote:

> As I've never seen (even on the wiki) an even halfway decent demonstration
> of using an ImageSegment, I'm providing on here.
>
>     hex := Browser allInstances first.
>
>     "let's check what we're saving to compare with later"
>     (hex buildWith: ToolBuilder default) openInWorld
>
>     exporting := (ImageSegment new copyFromRootsForExport: (Array with:
> hex)).
>
>     “let’s put it on disk”
>     exporting  writeForExport: 'browser.extSeg'.
>
>     "Quit your image without saving. Actually, to get the full effect go
> to http://ftp.squeak.org and get a fresh image of the same kind you
> exported with. Drag browser.extSeg into the new image directory"
>
>     “let’s pull it in from the disk”
>     importing := (FileDirectory default readOnlyFileNamed:
> 'browser.extSeg') fileInObjectAndCode.
>
>     "let's check what we imported to see if it's what we saved"
>     ((importing originalRoots first) buildWith: ToolBuilder default)
> openInWorld
>
> Chris
>
>
>
> On 2015-08-19 8:12 PM, David T. Lewis wrote:
>
>> I think that image segments are a worthwhile idea, regardless of whether
>> we are worried about support project saving and other image conversion
>> issues. Here is a post by Dan Ingalls from 1999 that summarizes the work
>> on image segments at that time:
>>
>>
>> http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>>
>> Dan also mentioned it in "the future of Squeak, 1999":
>>
>>    http://wiki.squeak.org/squeak/393
>>
>> It is quite clear that saving projects was a proof of concept to
>> illustrate
>> what might be done with image segments, but the overall motivation had
>> more
>> to do with exploring modularity and mechanisms for delivering minimal
>> images.
>>
>> Tim Rowledge added a comment to that effect here:
>>
>>
>> http://lists.squeakfoundation.org/pipermail/squeak-dev/1999-October/014604.html
>>
>> I will also note that Spur is not the first time we have needed to think
>> about making image segments work on a new image format. When Dan Ingalls
>> and
>> Ian Piumarta announced the first 64-bit Squeak image, they said "We may
>> ask for
>> help from the Squeak community in converting the remaining plugins, and
>> also
>> image segment code."
>>
>>
>> http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-August/081383.html
>>
>> Quite a few of the things that Dan and Ian asked for help on have since
>> been
>> done, but I don't think that image segment support for the 64-bit image
>> was
>> among them. With the arrival of Spur, and the upcoming Spur 64-bit image,
>> it
>> would be great if we can put some thought and effort into doing this
>> right.
>>
>> $0.02,
>>
>> Dave
>>
>>
>> On Wed, Aug 19, 2015 at 02:09:32PM -0700, Eliot Miranda wrote:
>>
>>>   Hi All, but especially Tobias,
>>>
>>>      as we know, ImageSegments are broken in Spur. They work
>>> occasionally,
>>> but when stressed the VM ends up crashing.  The fundamental issue is that
>>> the ImageSegment code makes assumptions about object ordering that Spur
>>> violates.  For example, here's ImageSegment's install:
>>>
>>> install
>>> "This operation retrieves the segment if necessary from file storage,
>>> installs it in memory, and replaces (using become:) all the root stubs
>>> with
>>> the reconstructed roots of the segment."
>>>
>>> | newRoots |
>>> state = #onFile ifTrue: [self readFromFile].
>>> state = #onFileWithSymbols ifTrue: [self readFromFileWithSymbols.
>>> endMarker := segment nextObject. "for enumeration of objects"
>>> endMarker == 0 ifTrue: [endMarker := 'End' clone]].
>>> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>>> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>>> state = #imported
>>> ifTrue: ["just came in from exported file"
>>> arrayOfRoots := newRoots]
>>> ifFalse: [
>>> arrayOfRoots elementsForwardIdentityTo: newRoots].
>>> state := #inactive.
>>> Beeper beepPrimitive
>>>
>>> So before the image segment bytes (the segment inst var) is loaded, the
>>> object after it is assigned to endMarker, and if there isn't an object
>>> after segment, a new object ('End' clone) is assigned to endMarker.
>>>
>>> This makes the assumption that objects are allocated in a strict order,
>>> and
>>> therefore endMarker will always be the object after segment.
>>>
>>> Loading the segment via "newRoots := self loadSegmentFrom: segment
>>> outPointers: outPointers" then turns segment into a zero-length WordArray
>>> and its contents into the objects loaded by the segment.  Therefore, in
>>> the
>>> V3 system, the objects loaded from segment can be enumerated starting at
>>> segment nextObject and repeating until endMarker is found:
>>>
>>> allObjectsDo: aBlock
>>> "Enumerate all objects that came from this segment.  NOTE this assumes
>>> that
>>> the segment was created (and extracted).  After the segment has been
>>> installed (install), this method allows you to enumerate its objects."
>>> | obj |
>>>
>>> endMarker == nil ifTrue: [
>>> ^ self error: 'Just extract and install, don''t writeToFile:'].
>>> segment size ~= 1 ifTrue: [
>>> ^ self error: 'Vestigial segment size must be 1 (version word)'].
>>>
>>> obj := segment nextObject.  "Start with the next object after the
>>> vestigial
>>> header"
>>> [obj == endMarker] whileFalse:  "Stop at the next object after the full
>>> segment"
>>> [aBlock value: obj.
>>> obj := obj nextObject].  "Step through the objects installed from the
>>> segment."
>>>
>>> Now, as written, this just won't work in Spur.
>>>
>>> a) the only place where there is any kind of stable order to objects is
>>> in
>>> oldSpace, so segment /has/ to be forced to old space to have any chance
>>> of
>>> its objects being in order when it gets converted from bytes to objects.
>>>
>>> b) 'End' clone will be in newSpace and so endMarker isn't reliable unless
>>> it was obtained by segment nextObject when segment was already in
>>> oldSpace.
>>>
>>> So it is perhaps possible to fix ImageSegments in Spur by forcing segment
>>> to oldSpace and being more careful with endMarker.  But I think there is
>>> a
>>> better way.
>>>
>>> If the set of objects the segment contains can be obtained some how then
>>> this set can be simply enumerated, not depending on nextObject.  The
>>> primitive has to answer the array of roots, so its result can't be
>>> changed
>>> to be the entire array.  But segment could be becomed into an Array of
>>> all
>>> the objects in segment prior to it being loaded, in which case the above
>>> would become
>>>
>>>
>>> install
>>> "This operation retrieves the segment if necessary from file storage,
>>> installs it in memory, and replaces (using become:) all the root stubs
>>> with
>>> the reconstructed roots of the segment."
>>>
>>> | newRoots |
>>> state = #onFile ifTrue: [self readFromFile].
>>> state = #onFileWithSymbols ifTrue:
>>> [self readFromFileWithSymbols].
>>> (state = #active) | (state = #imported) ifFalse: [self errorWrongState].
>>> newRoots := self loadSegmentFrom: segment outPointers: outPointers.
>>> state = #imported
>>> ifTrue: "just came in from exported file"
>>> [arrayOfRoots := newRoots]
>>> ifFalse:
>>> [arrayOfRoots elementsForwardIdentityTo: newRoots].
>>> state := #inactive.
>>> Beeper beepPrimitive
>>>
>>> allObjectsDo: aBlock
>>> "Enumerate all objects that came from this segment.  NOTE this assumes
>>> that
>>> the segment was created (and extracted).  After the segment has been
>>> installed (install), this method allows you to enumerate its objects."
>>> | obj |
>>>
>>> segment isArray ifFalse:
>>> [^ self error: 'Segment hasn''t been loaded?'].
>>>
>>> segment do: aBlock
>>>
>>> and the endMarker instVar would be deleted.
>>>
>>> I am willing and ready to modify the primitive to convert the segment
>>> correctly.  Who will volunteer to rewrite the image-level ImageSegment
>>> code
>>> to use the new primitive?
>>>
>>>
>>>
>>> On Wed, Aug 12, 2015 at 10:40 PM, Eliot Miranda <eliot.miranda at gmail.com
>>> >
>>> wrote:
>>>
>>> Hi Tobias,
>>>>
>>>> On Wed, Aug 12, 2015 at 10:18 PM, Tobias Pape <Das.Linux at gmx.de> wrote:
>>>>
>>>> Hi all
>>>>> On 13.08.2015, at 02:15, Eliot Miranda <eliot.miranda at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Tobias,
>>>>>>
>>>>>>     forget that. I found it.  THis right?
>>>>>>
>>>>>> Trunk test suite for Spur
>>>>>> Using existing cogspur r.3410
>>>>>> cp -r /var/lib/jenkins/workspace/Trunk/default/target/cogspur.r3410
>>>>>>
>>>>> /tmp/d20150812-28620-etbikj
>>>>>
>>>>>>    image test suite
>>>>>> VM:
>>>>>> /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
>>>>>> /tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak
>>>>>>
>>>>> -version
>>>>>
>>>>>> /var/lib/jenkins/workspace/Trunk/default/tests.st
>>>>>> spawning command 0 with timeout 1800 seconds:
>>>>>>
>>>>> "/tmp/d20150812-28620-etbikj/cogspur.r3410/cogspurlinuxht/bin/squeak"
>>>>> "-vm-sound-null" "-vm-display-null"
>>>>>
>>>>> "/var/lib/jenkins/workspace/Trunk/default/target/SpurPostTestTrunkImage.image"
>>>>> "../tests.st"
>>>>>
>>>>>> (Command started with PID 28643)
>>>>>> 2015-08-12T23:31:48.17+01:00: Loading Hudson build tools... from
>>>>>>
>>>>> /var/lib/jenkins/workspace/Trunk/default/target/HudsonBuildTools.st
>>>>>
>>>>>> 2015-08-12T23:31:48.388+01:00: Running tests...
>>>>>> setsockopt: Protocol not available
>>>>>> setsockopt: Protocol not available
>>>>>> 28646:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
>>>>>>
>>>>> protocol:s23_clnt.c:612:
>>>>>
>>>>>> Recursive not understood error encountered
>>>>>>
>>>>>>
>>>>>> I bet this is ImageSegment related.
>>>>>>
>>>>> It sure is.
>>>>>
>>>>>          BitmapStreamTests>testMatrixTransform2x3WithImageSegment
>>>>> sends
>>>>>          BitmapStreamTests>validateImageSegment
>>>>>
>>>>>   Basically the ImageSegment code has been working in Spur on a hope
>>>> and a
>>>> prayer.  The code assumes objects are allocated in chronological order
>>>> and
>>>> that this order is preserved, not so with Spur.  So post image segment
>>>> loading someObject/nextObject is used to enumerate the objects loaded.
>>>> This can't work reliably in Spur.  I *think* (ok I hope) that I've
>>>> implemented the segment load primitive in Spur to answer an Array of the
>>>> objects loaded, so that these can be explicitly enumerated.
>>>>
>>>> So the job is a) to check that I have indeed implemented the primitive
>>>> to
>>>> do this and b) to rewrite the image segment loading code in the light of
>>>> this.
>>>>
>>>> David, this is an example of something that isn't back portable and
>>>> should
>>>> not be back-ported.
>>>>
>>>> The most strange thing is that I cannot reproduce this on my Mac???
>>>>
>>>>> There the test does not crash the image???
>>>>>
>>>>> Well then it may be a signed/unsigned bug in image loading instead.  On
>>>> linux the image typically gets loaded quite high in the address space
>>>> and
>>>> so a good portion of the heap ends up above 0x7FFFFFFF, or negative
>>>> territory if misinterpreted as signed integers.  On Mac and Windows the
>>>> image tends to get loaded quite low and so has to be pretty big to fall
>>>> foul of signedness issues.
>>>>
>>>>
>>>> Busy right now but will check tomorrow.
>>>>
>>>>
>>>> Best regards
>>>>>          -Tobias
>>>>>
>>>>>
>>>>> On Wed, Aug 12, 2015 at 5:11 PM, Eliot Miranda <
>>>>>> eliot.miranda at gmail.com>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>> Hi Tobias,
>>>>>>
>>>>>> On Wed, Aug 12, 2015 at 12:49 PM, Tobias Pape <Das.Linux at gmx.de>
>>>>>> wrote:
>>>>>>
>>>>>> On 12.08.2015, at 20:55, Eliot Miranda <eliot.miranda at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi All,
>>>>>>>
>>>>>>>
>>>>>>>      Fabio's kindly done most of the changes.  But some questions
>>>>>>>
>>>>>> remain for more general discussion.  See below.
>>>>>
>>>>>> Fabio,  thanks so much for doing this, and so quickly!
>>>>>>>
>>>>>>> On Tue, Aug 11, 2015 at 8:51 PM, Eliot Miranda <
>>>>>>>
>>>>>> eliot.miranda at gmail.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>>
>>>>>>>      who will update http://squeak.org/downloads/ to include the 5.0
>>>>>>>
>>>>>> release?
>>>>>
>>>>>> a) I suggest that the 5.0 all-in-one have a line in the left-hand
>>>>>>>
>>>>>> table that includes the 4.6 release and that it precede the 4.6
>>>>> release in
>>>>> the list.
>>>>>
>>>>>> Done.
>>>>>>>
>>>>>>>
>>>>>>> b) the Trunk link points to TrunkImage.zip which contains a non-Spur
>>>>>>>
>>>>>> image.
>>>>>
>>>>>> So this is the biggie.  What should we do?
>>>>>>>
>>>>>>> - add a Trunk 5.0 build with a different link?  (Noooooo)
>>>>>>>
>>>>>>> - change  http://build.squeak.org/job/SqueakTrunk to build what it
>>>>>>>
>>>>>> says (Yessss please, who can do this?)
>>>>>
>>>>>> Done: build.squeak.org/job/Trunk/
>>>>>> But as I said several times, Spur/trunk test just crash for month???
>>>>>>
>>>>>> can you point me to the crash?  I'm looking at
>>>>>>
>>>>> http://build.squeak.org/job/Trunk/default/lastBuild/#showFailuresLink
>>>>> and see 8 test failures but no crash.
>>>>>
>>>>>>
>>>>>> - add a Squeak 4.6 build job?  Is that even worth it any more
>>>>>>>
>>>>>> considering 4.6 is released and stable?  If it is then what should it
>>>>> build?  David?
>>>>>
>>>>>>
>>>>>>> c) Spur VMs need to be linked to by an added line in the Virtual
>>>>>>>
>>>>>> Machines list.
>>>>>
>>>>>> Not applicable.  the VM links point to the root of my site so they
>>>>>>>
>>>>>> effectively point to both old and Spur VMs.
>>>>>
>>>>>>
>>>>>>> d) Spur imagers need to be included in the Image and Changes list
>>>>>>>
>>>>>> under Custom Installation
>>>>>
>>>>>> Done.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> e) The SqueakV50.sources file also needs to be included in the
>>>>>>>
>>>>>> Current Sources list.
>>>>>
>>>>>> Done.
>>>>>>>
>>>>>>>
>>>>>>> f) we could start a History list for the 5.0 line
>>>>>>>
>>>>>>> This is probably quite a big reorg on the page and not urgent.
>>>>>>>
>>>>>>>
>>>>>>> So there are a few things to fix before 5.0 is freely downloadable.
>>>>>>>
>>>>>>> On Tue, Aug 11, 2015 at 8:23 PM, Chris Muller <ma.chris.m at gmail.com>
>>>>>>>
>>>>>> wrote:
>>>>>
>>>>>> In the 17 months since Squeak 4.5 was released, a huge development
>>>>>>> effort took place to create the next generation virtual-machine for
>>>>>>> the Squeak / Pharo / Newspeak family of programming systems.  Squeak
>>>>>>> is the modern incarnation of the Smalltalk-80 programming environment
>>>>>>> originally developed at the Xerox PARC.
>>>>>>>
>>>>>>> "Squeak 5" introduces this new VM and associated new memory model,
>>>>>>> collectively referred to as "Spur".  Presented [1] by Eliot Miranda
>>>>>>> and Cl??ment B??ra at the 2015 International Symposium on Memory
>>>>>>> Management, this new VM affords Squeak applications a significant
>>>>>>> boost in performance and memory management.  Among other
>>>>>>> optimizations, the #become operation no longer requires a memory
>>>>>>> scan.
>>>>>>> Object pinning and ephemerons are also now supported.  The release
>>>>>>> notes [2] provide more details.
>>>>>>>
>>>>>>> The new memory model requires a new image file format.  Although this
>>>>>>> new format results in about a 15% increased memory requirement for
>>>>>>> the
>>>>>>> same number of 4.x objects, a new segmented heap allows memory to be
>>>>>>> given back to the OS when its no longer needed, a great benefit for
>>>>>>> application servers.
>>>>>>>
>>>>>>> As forward compatibility is as important to the Squeak community as
>>>>>>> backward compatibility, Squeak 5 is delivers an image with identical
>>>>>>> content as the recent 4.6 release.  Although this new Squeak 5 VM
>>>>>>> cannot open images saved under the prior 4.x Cog format, objects and
>>>>>>> code can be easily exported from the 4.x image and then imported into
>>>>>>> Squeak 5.  Applications whose code runs strictly above the Smalltalk
>>>>>>> meta layer will prove remarkably compatible with the new format, most
>>>>>>> applications will require no changes whatsotever.
>>>>>>>
>>>>>>> Squeak 5 is the result of monumental effort by a tiny group of very
>>>>>>> talented people, but its also just the beginning of yet a new effort;
>>>>>>> Spur is just a stepping stone to a more ambitious goals planned over
>>>>>>> the next five years.
>>>>>>>
>>>>>>> [1] -- A Partial Read Barrier for Efficient Support of Live
>>>>>>> Object-oriented Programming
>>>>>>>
>>>>>>>
>>>>> http://conf.researchr.org/event/ismm-2015/ismm-2015-papers-a-partial-read-barrier-for-efficient-support-of-live-object-oriented-programming
>>>>>
>>>>>> [2] -- Squeak 5 Release Notes
>>>>>>> http://wiki.squeak.org/squeak/6207
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> _,,,^..^,,,_
>>>> best, Eliot
>>>>
>>>>
>>>
>>> --
>>> _,,,^..^,,,_
>>> best, Eliot
>>>
>>
>>
>
>


-- 
Mariano
http://marianopeck.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20150819/2f603c0f/attachment-0001.htm