[Vm-dev] I think I've found a bug in the 64-bit Squeak VM memory compactor

ungar at mac.com ungar at mac.com
Tue Jun 15 06:31:53 UTC 2010


Hello Dave,

You are welcome. I am indeed reading  in a snapshot with 32-bit oops into the stock ST Mac OSX VM I downloaded from the website. The file name is: Squeak 64-32 5.4b2. I don't know what its setting of SQ_FAKE_MEMORY_OFFSET is, since I am not building it, just using it. I do know that the first word of the heap is called "0" when running in 64-bit mode. (It is a fat binary that will run on an Intel CPU either in 32 or 64 bit mode.)

I am not (perhaps yet) working with a 64-bit oops size, but may do so in the future, and if so, your pointer will be helpful. Thank you!

Do you know who might be in a position to look at the bug I found and decide what is the right thing to do for the Squeak community? I am trying to figure out who "holds the baton" for that system.

Thank you,

 - David


On Jun 14, 2010, at 12:06 PM, David T. Lewis wrote:

> 
> Hello David and thank you for reporting this.
> 
> I am assuming that you are working with a 32-bit word size object
> memory (not the Squeak 64-bit object memory), and that "64 bit VM"
> refers to compiling for 64-bit systems (64-bit machine pointer size).
> If so, the difference that you are seeing between the 32-bit and 64-bit
> version may be due to the setting of SQ_FAKE_MEMORY_OFFSET in the
> platforms/Cross/vm/sqMemoryAccess.h header file. This is set to a non-zero
> value for the the combination of 32-bit object work and 64-bit host machine,
> specifically to help flush out bugs like the one you have found here.
> 
> I suspect that if you set SQ_FAKE_MEMORY_OFFSET to 0, the problem may
> appear to go away.
> 
> If you are also working with 64 bit object memories, then you will want
> to be aware of the (probably unrelated) issue that is documented at
> <http://bugs.squeak.org/view.php?id=7455>. The bug is harmless at the
> moment, but if you are making any changes to the image snapshot file
> header you will want to keep it in mind. The summary is:
> 
>  7455: VM overwrites extraVmMemory value with junk on 64 bit image 
>  A 64 bit image (wordSize 8) is currently using a 128 byte image file header,
>  with all the integer values in the header stored as 8 bit ints. The VM is
>  writing the image data at position 64 rather than 128, resulting the the
>  value of extraVmMemory being overwritten with garbage. ImageTracer64
>  produces the correct 128 byte offset, but this is undone by the VM after
>  the first image save.
> 
> Dave
> 
> On Mon, Jun 14, 2010 at 10:00:48AM -0700, ungar at mac.com wrote:
>> 
>> Friends, Romans, and SqueakVMers,
>> 
>> As some of you know, I am working on my own Squeak VM, and I recently ran into a strange situation where the 64-bit Squeak VM would loop forever reading one of my snapshots. The 32-bit version--that is, just running it in 32-bit mode--worked fine. I spent some time tracking this down, and believe that the problem was caused by a bug in the 64-bit Squeak VM GC code that is only excited when the first word in the heap is the start of a free chunk. I don't know where I should be sending this report, so I'm sending it to you, in hopes that you may be able to forward it to the right place and/or people.
>> 
>> 
>> Here is the explanation:
>> 
>> 
>> 
>> Since my object format uses extra words per object preceding the standard ST header, when I write out a snapshot, I write those words out as "Free chunks", the official ST way of saying there are X words not part of any object.
>> In the 32-bit version of the real Squeak VM, Oops are (tagged) addresses of the target objects.
>> The 64-bit version of the real Squeak VM, each 32-bit  oop in the image is added to an offset to product the (64-bit) address of the object. Thus, oops in the image are relative to the start of the heap.
>> 
>> The sweep phase of the Squeak VM GC is this:
>> 
>> 
>>> sweepPhase
>>> 	"Sweep memory from youngStart through the end of memory. Free all
>>> 	inaccessible objects and coalesce adjacent free chunks. Clear the mark
>>> 	bits of accessible objects. Compute the starting point for the first pass of
>>> 	incremental compaction (compStart). Return the number of surviving
>>> 	objects. "
>>> 	"Details: Each time a non-free object is encountered, decrement the
>>> 	number of available forward table entries. If all entries are spoken for
>>> 	(i.e., entriesAvailable reaches zero), set compStart to the last free
>>> 	chunk before that object or, if there is no free chunk before the given
>>> 	object, the first free chunk after it. Thus, at the end of the sweep
>>> 	phase, compStart through compEnd spans the highest collection of
>>> 	non-free objects that can be accomodated by the forwarding table. This
>>> 	information is used by the first pass of incremental compaction to
>>> 	ensure that space is initially freed at the end of memory. Note that
>>> 	there should always be at least one free chunk--the one at the end of
>>> 	the heap."
>>> 	| entriesAvailable survivors freeChunk firstFree oop oopHeader oopHeaderType hdrBytes oopSize freeChunkSize endOfMemoryLocal |
>>> 	self inline: false.
>>> 	self var: #oop type: 'usqInt'.
>>> 	self var: #endOfMemoryLocal type: 'usqInt'.
>>> 	entriesAvailable := self fwdTableInit: self bytesPerWord * 2.
>>> 	survivors := 0.
>>> 	freeChunk := nil.
>>> 	firstFree := nil.
>>> 	"will be updated later"
>>> 	endOfMemoryLocal := endOfMemory.
>>> 	oop := self oopFromChunk: youngStart.
>>> 	[oop<  endOfMemoryLocal]
>>> 		whileTrue: ["get oop's header, header type, size, and header size"
>>> 			statSweepCount := statSweepCount + 1.
>>> 			oopHeader := self baseHeader: oop.
>>> 			oopHeaderType := oopHeader bitAnd: TypeMask.
>>> 			hdrBytes := headerTypeBytes at: oopHeaderType.
>>> 			(oopHeaderType bitAnd: 1) = 1
>>> 				ifTrue: [oopSize := oopHeader bitAnd: self sizeMask]
>>> 				ifFalse: [oopHeaderType = HeaderTypeSizeAndClass
>>> 						ifTrue: [oopSize := (self sizeHeader: oop) bitAnd: self longSizeMask]
>>> 						ifFalse: ["free chunk" oopSize := oopHeader bitAnd: self longSizeMask]].
>>> 			(oopHeader bitAnd: self markBit) = 0
>>> 				ifTrue: ["object is not marked; free it"
>>> 					"<-- Finalization support: We need to mark each oop chunk as free -->"
>>> 					self longAt: oop - hdrBytes put: HeaderTypeFree.
>>> 					freeChunk ~= nil
>>> 						ifTrue: ["enlarge current free chunk to include this oop"
>>> 							freeChunkSize := freeChunkSize + oopSize + hdrBytes]
>>> 						ifFalse: ["start a new free chunk"
>>> 							freeChunk := oop - hdrBytes.
>>> 							"chunk may start 4 or 8 bytes before oop"
>>> 							freeChunkSize := oopSize + (oop - freeChunk).
>>> 							"adjust size for possible extra header bytes"
>>> 							firstFree = nil ifTrue: [firstFree := freeChunk]]]
>>> 				ifFalse: ["object is marked; clear its mark bit and possibly adjust
>>> 					the compaction start"
>>> 					self longAt: oop put: (oopHeader bitAnd: self allButMarkBit).
>>> 					"<-- Finalization support: Check if we're running about a weak class -->"
>>> 					(self isWeakNonInt: oop) ifTrue: [self finalizeReference: oop].
>>> 					entriesAvailable>  0
>>> 						ifTrue: [entriesAvailable := entriesAvailable - 1]
>>> 						ifFalse: ["start compaction at the last free chunk before this object"
>>> 							firstFree := freeChunk].
>>> 					freeChunk ~= nil "BUG cannot handle free start--dmu! ***********************************************"
>>> 						ifTrue: ["record the size of the last free chunk"
>>> 							self longAt: freeChunk put: ((freeChunkSize bitAnd: self longSizeMask) bitOr: HeaderTypeFree).
>>> 							freeChunk := nil].
>>> 					survivors := survivors + 1].
>>> 			oop := self oopFromChunk: oop + oopSize].
>>> 	freeChunk ~= nil
>>> 		ifTrue: ["record size of final free chunk"
>>> 			self longAt: freeChunk put: ((freeChunkSize bitAnd: self longSizeMask) bitOr: HeaderTypeFree)].
>>> 	oop = endOfMemory
>>> 		ifFalse: [self error: 'sweep failed to find exact end of memory'].
>>> 	firstFree = nil
>>> 		ifTrue: [self error: 'expected to find at least one free object']
>>> 		ifFalse: [compStart := firstFree].
>>> 
>>> 	^ survivors
>>> 
>> 
>> Notice the line with the comment I added, it has a lot of asterisks. It is trying to go back to the last free chunk found and update its length, now that its found non-free chunk.
>> But, the test works by initializing the "freeChunk" variable to zero (called "nil" in this code)!!
>> If the first free chunk is at the start of the heap, it's "oop" will be zero in the 64-bit case, and it's length will NEVER be initialized.
>> As a result, when the compaction algorithm scans the heap, it gets stuck on this zero-length free chunk and loops forever.
>> (I think the other nil tests are troublemakers, too, BTW).
>> 
>> The workaround for my VM is simply to skip those first free words, so that the first word in the heap is a non-free object. (The snapshot code does a GC so all real free objects are gone by this time).
>> 
>> However, you might want to redo the tests so the algorithm tolerates a free chunk at the start of the heap.
>> 
>> Thank you for all your efforts!
>> 
>> - David
>> 
>> 
>> 
>> 



More information about the Vm-dev mailing list