[Vm-dev] [squeak-dev] Error in ImageSegment primitive?

Tue May 9 22:07:29 UTC 2017

NativeImageSegment sounds good.

(I had renamed "my" class to OldImageSegment but ran into issues with old
code breaking. Better to start clean)

- Bert -

On Tue 9. May 2017 at 23:05, Eliot Miranda <eliot.miranda at gmail.com> wrote:

>
> Hi Max, Bert,
>
>     to integrate the below with Bert's rewriting of the importer, I'm
> thinking of renaming ImageSegment to NativeImageSegment.  Hopefully we can
> get the native and the all-in-Smalltalk code to coexist.  Is
> NativeImageSegment a good name?  Do you prefer e.g. VMImageSegment?
> SpurImageSegment?
>
> And we still have to handle fixing up of loaded objects.  The load
> primitive answers the array of roots.  We'd actually like an array of all
> loaded objects.  Maybe I should add a variant of the load primitive that
> does this.  Are you, Max, up to rewriting the post-load object mixup to
> avoid nextObject?
>
> On Tue, May 9, 2017 at 10:15 AM, Max Leske <maxleske at gmail.com> wrote:
>
>>
>> [ Reposting with correct subject. Sorry! ]
>>
>>
>> On 9 May 2017, at 17:46, vm-dev-request at lists.squeakfoundation.org wrote:
>>
>> Hi Max,
>>
>> On May 9, 2017, at 7:30 AM, Max Leske <maxleske at gmail.com> wrote:
>>
>> Hi Eliot,
>>
>>
>> On 9 May 2017, at 14:32, vm-dev-request at lists.squeakfoundation.org wrote:
>>
>> Hi Max,
>>
>> On Sun, May 7, 2017 at 4:24 AM, Max Leske <maxleske at gmail.com> wrote:
>>
>> Hi,
>>
>> I'm trying to store an image segment with the latest pharo.cog.spur VM (32
>> bits) but keep failing. The segment should produce a file of around 60 MB.
>> With an old V3 VM this is no problem at all. There, the
>> WordArrayForSegment
>> instance has a size of 4094179 but with the new VM I always run out of
>> space because the primitive returns nil and, therefore, the word array
>> size
>> is constantly being increased.
>>
>>
>> It's been a while since I wrote this code so my understanding has been
>> coming back in fits and starts.  The limitation on the use of the hash
>> bits
>> field is in referring to "out pointers", objects that the saved segment
>> refers to, not on objects internal to the segment.  So I think it's
>> fixable.
>>
>> The hash field is used to map from an object in the heap to its object in
>> the segment.  Right now the mapping is from hash (22 bits) to location in
>> the segment / 8, and so limits the size of the segment to 500kb.  If an
>> extra level of indirection was added so that hash maps to index in an
>> array
>> of oops, then the segment could contain up to 4m objects and I think
>> that'll be large enough for your use.
>>
>> If that's still not enough then the al;goriqhm will have to be rewritten
>> to
>> use the first field of the object in the heap to point to its location in
>> the segment, and the first field saved alongside.
>>
>> So let me know.  Would you be happy with a fix that provides up to 4m
>> objects per segment or would you want to wait for something with a much
>> higher limit?
>>
>>
>> I've sampled 10000 classes and 10000 model specific classes at random and
>> got a median #byteSizeOfInstance of 60 bytes. For 4m objects that would
>> mean an upper limit of 240 MB file size. That is enough for me at the
>> moment, yes. I currently need around 120 MB (with a safety margin of 15 MB).
>> Not having to watch out for that limit would be nice but my priority is
>> to be able to create and read segments.
>>
>>
>> Agreed, but that's a bigger rewrite.  It would have in common some of the
>> approach taken with the new compactor which also uses the first field, in
>> its case to point to eventual location, while saving the contents of the
>> first field in an array off to the side.
>>
>> (In Spur all objects, including zero-sized ones, have at least one field
>> so that they can be converted into a forwarding pointer).
>>
>> How much time do you think it will take you to make the change? I just
>> need a rough idea so I can plan my work around that.
>>
>>
>> It shouldn't take more than a day; two at the most.  You can help me by
>> setting up a test case (although perhaps I can hack up a quick binary tree,
>> so that might not help as much as I expect).
>>
>>
>> I've tried to come up with a test case and have one that doesn't fail...
>> Not sure what I'm missing. The test reports the sizes of the segment and
>> its out pointers. I'm including it here, maybe you can work from that. I'm
>> also appending my hacked version of ImageSegment for Pharo 6 (mainly use of
>> FileSystem instead of FileDirectory).
>>
>> I hope that helps.
>>
>> Cheers,
>> Max
>>
>>
>> The primitive uses a routine that answers the objects to be written to
>> the segment in an array.  The primitive needs to be extended with a word
>> array, let's call it the mapArray, as big as this to hold the indirection
>> from hash field to index in the mapArray in which are held the pointers to
>> locations in the segment data.
>>
>> It might be just as easy/difficult to use a savedFirstFields array (also
>> one element per object to be copied into the array) and point to location
>> from first field, which would eliminate the upper limit immediately.  I'll
>> mull this over while at the DMV; a chore I have to do today.
>>
>> Thanks for your help!
>>
>> Cheers,
>> Max
>>
>>
>>
>> I've built a debug VM and am stepping through the code but I don't have a
>> clear understanding of everything that's happening. The failure happens on
>> line 46626 of gcc3x-cointerp.c:
>>
>> newOop = (copy - segStart) / 8;
>> if (newOop > (identityHashHalfWordMask())) {
>>       return PrimErrLimitExceeded;  // <--------------- failure
>> }
>>
>> What I don't understand, for example, is why "newOop" is checked against
>> "identityHashHalfWordMask()" and not against the segment end ("endSeg").
>> Here's a list of the current values of the variables upon failure:
>>
>> objOop  sqInt   180812096
>> segAddr sqInt   494731288
>> segStart        sqInt   461176856
>> endSeg  sqInt   815841432
>> bodySize        usqInt  64
>> contextSize     sqInt   335672448
>> copy    sqInt   494731288
>> hash    sqInt   0
>> hash1   sqInt   4194302
>> i       sqInt   833574680
>> iLimiT  sqInt   833574688
>> methodHeader    sqInt   1193471
>> newOop  sqInt   4194304
>> numMediatedSlots        sqInt   833574688
>> numSlots        usqInt  14
>> oop     sqInt   142640272
>>
>> As you can see, "endSeg" would be more than large enough to hold the
>> object. Is it possible that there's an error here?
>>
>> Cheers,
>> Max
>>
>>
>>
>>
>> --
>> _,,,^..^,,,_
>> best, Eliot
>>
>>
>>
>>
>>
>>
>
>
> --
> _,,,^..^,,,_
> best, Eliot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20170509/e0e62cb4/attachment.html>