[Vm-dev] I think I've found a bug in the 64-bit Squeak VM memory
compactor
David T. Lewis
lewis at mail.msen.com
Mon Jun 14 19:06:39 UTC 2010
Hello David and thank you for reporting this.
I am assuming that you are working with a 32-bit word size object
memory (not the Squeak 64-bit object memory), and that "64 bit VM"
refers to compiling for 64-bit systems (64-bit machine pointer size).
If so, the difference that you are seeing between the 32-bit and 64-bit
version may be due to the setting of SQ_FAKE_MEMORY_OFFSET in the
platforms/Cross/vm/sqMemoryAccess.h header file. This is set to a non-zero
value for the the combination of 32-bit object work and 64-bit host machine,
specifically to help flush out bugs like the one you have found here.
I suspect that if you set SQ_FAKE_MEMORY_OFFSET to 0, the problem may
appear to go away.
If you are also working with 64 bit object memories, then you will want
to be aware of the (probably unrelated) issue that is documented at
<http://bugs.squeak.org/view.php?id=7455>. The bug is harmless at the
moment, but if you are making any changes to the image snapshot file
header you will want to keep it in mind. The summary is:
7455: VM overwrites extraVmMemory value with junk on 64 bit image
A 64 bit image (wordSize 8) is currently using a 128 byte image file header,
with all the integer values in the header stored as 8 bit ints. The VM is
writing the image data at position 64 rather than 128, resulting the the
value of extraVmMemory being overwritten with garbage. ImageTracer64
produces the correct 128 byte offset, but this is undone by the VM after
the first image save.
Dave
On Mon, Jun 14, 2010 at 10:00:48AM -0700, ungar at mac.com wrote:
>
> Friends, Romans, and SqueakVMers,
>
> As some of you know, I am working on my own Squeak VM, and I recently ran into a strange situation where the 64-bit Squeak VM would loop forever reading one of my snapshots. The 32-bit version--that is, just running it in 32-bit mode--worked fine. I spent some time tracking this down, and believe that the problem was caused by a bug in the 64-bit Squeak VM GC code that is only excited when the first word in the heap is the start of a free chunk. I don't know where I should be sending this report, so I'm sending it to you, in hopes that you may be able to forward it to the right place and/or people.
>
>
> Here is the explanation:
>
>
>
> Since my object format uses extra words per object preceding the standard ST header, when I write out a snapshot, I write those words out as "Free chunks", the official ST way of saying there are X words not part of any object.
> In the 32-bit version of the real Squeak VM, Oops are (tagged) addresses of the target objects.
> The 64-bit version of the real Squeak VM, each 32-bit oop in the image is added to an offset to product the (64-bit) address of the object. Thus, oops in the image are relative to the start of the heap.
>
> The sweep phase of the Squeak VM GC is this:
>
>
> > sweepPhase
> > "Sweep memory from youngStart through the end of memory. Free all
> > inaccessible objects and coalesce adjacent free chunks. Clear the mark
> > bits of accessible objects. Compute the starting point for the first pass of
> > incremental compaction (compStart). Return the number of surviving
> > objects. "
> > "Details: Each time a non-free object is encountered, decrement the
> > number of available forward table entries. If all entries are spoken for
> > (i.e., entriesAvailable reaches zero), set compStart to the last free
> > chunk before that object or, if there is no free chunk before the given
> > object, the first free chunk after it. Thus, at the end of the sweep
> > phase, compStart through compEnd spans the highest collection of
> > non-free objects that can be accomodated by the forwarding table. This
> > information is used by the first pass of incremental compaction to
> > ensure that space is initially freed at the end of memory. Note that
> > there should always be at least one free chunk--the one at the end of
> > the heap."
> > | entriesAvailable survivors freeChunk firstFree oop oopHeader oopHeaderType hdrBytes oopSize freeChunkSize endOfMemoryLocal |
> > self inline: false.
> > self var: #oop type: 'usqInt'.
> > self var: #endOfMemoryLocal type: 'usqInt'.
> > entriesAvailable := self fwdTableInit: self bytesPerWord * 2.
> > survivors := 0.
> > freeChunk := nil.
> > firstFree := nil.
> > "will be updated later"
> > endOfMemoryLocal := endOfMemory.
> > oop := self oopFromChunk: youngStart.
> > [oop< endOfMemoryLocal]
> > whileTrue: ["get oop's header, header type, size, and header size"
> > statSweepCount := statSweepCount + 1.
> > oopHeader := self baseHeader: oop.
> > oopHeaderType := oopHeader bitAnd: TypeMask.
> > hdrBytes := headerTypeBytes at: oopHeaderType.
> > (oopHeaderType bitAnd: 1) = 1
> > ifTrue: [oopSize := oopHeader bitAnd: self sizeMask]
> > ifFalse: [oopHeaderType = HeaderTypeSizeAndClass
> > ifTrue: [oopSize := (self sizeHeader: oop) bitAnd: self longSizeMask]
> > ifFalse: ["free chunk" oopSize := oopHeader bitAnd: self longSizeMask]].
> > (oopHeader bitAnd: self markBit) = 0
> > ifTrue: ["object is not marked; free it"
> > "<-- Finalization support: We need to mark each oop chunk as free -->"
> > self longAt: oop - hdrBytes put: HeaderTypeFree.
> > freeChunk ~= nil
> > ifTrue: ["enlarge current free chunk to include this oop"
> > freeChunkSize := freeChunkSize + oopSize + hdrBytes]
> > ifFalse: ["start a new free chunk"
> > freeChunk := oop - hdrBytes.
> > "chunk may start 4 or 8 bytes before oop"
> > freeChunkSize := oopSize + (oop - freeChunk).
> > "adjust size for possible extra header bytes"
> > firstFree = nil ifTrue: [firstFree := freeChunk]]]
> > ifFalse: ["object is marked; clear its mark bit and possibly adjust
> > the compaction start"
> > self longAt: oop put: (oopHeader bitAnd: self allButMarkBit).
> > "<-- Finalization support: Check if we're running about a weak class -->"
> > (self isWeakNonInt: oop) ifTrue: [self finalizeReference: oop].
> > entriesAvailable> 0
> > ifTrue: [entriesAvailable := entriesAvailable - 1]
> > ifFalse: ["start compaction at the last free chunk before this object"
> > firstFree := freeChunk].
> > freeChunk ~= nil "BUG cannot handle free start--dmu! ***********************************************"
> > ifTrue: ["record the size of the last free chunk"
> > self longAt: freeChunk put: ((freeChunkSize bitAnd: self longSizeMask) bitOr: HeaderTypeFree).
> > freeChunk := nil].
> > survivors := survivors + 1].
> > oop := self oopFromChunk: oop + oopSize].
> > freeChunk ~= nil
> > ifTrue: ["record size of final free chunk"
> > self longAt: freeChunk put: ((freeChunkSize bitAnd: self longSizeMask) bitOr: HeaderTypeFree)].
> > oop = endOfMemory
> > ifFalse: [self error: 'sweep failed to find exact end of memory'].
> > firstFree = nil
> > ifTrue: [self error: 'expected to find at least one free object']
> > ifFalse: [compStart := firstFree].
> >
> > ^ survivors
> >
>
> Notice the line with the comment I added, it has a lot of asterisks. It is trying to go back to the last free chunk found and update its length, now that its found non-free chunk.
> But, the test works by initializing the "freeChunk" variable to zero (called "nil" in this code)!!
> If the first free chunk is at the start of the heap, it's "oop" will be zero in the 64-bit case, and it's length will NEVER be initialized.
> As a result, when the compaction algorithm scans the heap, it gets stuck on this zero-length free chunk and loops forever.
> (I think the other nil tests are troublemakers, too, BTW).
>
> The workaround for my VM is simply to skip those first free words, so that the first word in the heap is a non-free object. (The snapshot code does a GC so all real free objects are gone by this time).
>
> However, you might want to redo the tests so the algorithm tolerates a free chunk at the start of the heap.
>
> Thank you for all your efforts!
>
> - David
>
>
>
>
More information about the Vm-dev
mailing list