[Vm-dev] VM Maker: Cog-eem.328.mcz

commits at source.squeak.org commits at source.squeak.org
Tue Aug 16 02:50:49 UTC 2016


Eliot Miranda uploaded a new version of Cog to project VM Maker:
http://source.squeak.org/VMMaker/Cog-eem.328.mcz

==================== Summary ====================

Name: Cog-eem.328
Author: eem
Time: 15 August 2016, 7:50:11.265382 pm
UUID: 09c34c1e-71c5-4a9f-870c-e441d6d24011
Ancestors: Cog-eem.327

Provide Spur32BitPreen to opreen 32-bit images, removing the excessive free space left by Spur's current poor compaction algorithm.

=============== Diff against Cog-eem.327 ===============

Item was added:
+ SimulatorHarness subclass: #Spur32BitPreen
+ 	instanceVariableNames: 'oldHeap newHeap map reverseMap oldInterpreter newInterpreter imageHeaderFlags savedWindowSize'
+ 	classVariableNames: ''
+ 	poolDictionaries: 'VMObjectIndices'
+ 	category: 'Cog-Bootstrapping'!
+ 
+ !Spur32BitPreen commentStamp: 'eem 8/15/2016 19:49' prior: 0!
+ A Spur32BitPreen is a simple image rewriter for 32-bit Spru images that eliminates free space and hence shrinks the preened image.  Use via
+ 	Spur32BitPreen new preenImage: 'spur'
+ which will produce spur-preen.image and spur-preen.changes from spur.image and spur.changes.
+ 
+ Instance Variables
+ 	imageHeaderFlags:		<Integer>
+ 	map:					<Dictionary>
+ 	newHeap:				<Spur32BitMMLESimulator>
+ 	newInterpreter:			<StackInterpreterSimulatorLSB>
+ 	oldHeap:				<Spur32BitMMLESimulator>
+ 	oldInterpreter:			<StackInterpreterSimulatorLSB>
+ 	reverseMap:			<Dictionary>
+ 	savedWindowSize:		<Integer>
+ 
+ imageHeaderFlags
+ 	- flags word in image header
+ 
+ map
+ 	- map from oops in old image to oops in new image
+ 
+ newHeap
+ 	- the preened heap
+ 
+ newInterpreter
+ 	- the interpreter wrapping the preened heap
+ 
+ oldHeap
+ 	- the heap to be preened
+ 
+ oldInterpreter
+ 	- the interpreter wrapping the heap to be preened
+ 
+ reverseMap
+ 	- map from oops in new image to oops in old image
+ 
+ savedWindowSize
+ 	- screen size word in mage header
+ !

Item was added:
+ ----- Method: Spur32BitPreen>>clone: (in category 'bootstrap image') -----
+ clone: oldObj
+ 	| newObj hash |
+ 	newObj := newHeap
+ 				allocateSlots: (oldHeap numSlotsOf: oldObj)
+ 				format: (oldHeap formatOf: oldObj)
+ 				classIndex: (oldHeap classIndexOf: oldObj).
+ 	(hash := oldHeap rawHashBitsOf: oldObj) ~= 0 ifTrue:
+ 		[newHeap setHashBitsOf: newObj to: hash].
+ 	(oldHeap isImmutable: oldObj) ifTrue:
+ 		[newHeap setIsImmutableOf: newObj to: true].
+ 	(oldHeap isPinned: oldObj) ifTrue:
+ 		[newHeap setIsPinnedOf: newObj to: true].
+ 	self deny: (oldHeap isRemembered: oldObj).
+ 	self deny: (oldHeap isMarked: oldObj).
+ 	self deny: (oldHeap isGrey: oldObj).
+ 	reverseMap at: newObj put: oldObj.
+ 	^map at: oldObj put: newObj!

Item was added:
+ ----- Method: Spur32BitPreen>>cloneFreeLists: (in category 'bootstrap image') -----
+ cloneFreeLists: oldObj
+ 	| newObj |
+ 	newObj := newHeap
+ 				allocateSlots: newHeap numFreeLists
+ 				format: newHeap wordIndexableFormat
+ 				classIndex: newHeap wordSizeClassIndexPun.
+ 	reverseMap at: newObj put: oldObj.
+ 	^map at: oldObj put: newObj!

Item was added:
+ ----- Method: Spur32BitPreen>>cloneObjects (in category 'bootstrap image') -----
+ cloneObjects
+ 	"Clone all normal objects.  Of hidden objects only clone the freeLists object and
+ 	 the classTableRoot and class table pages. In particular, dont clone objStacks.
+ 	 The refs to the objStacks are nilled out in fillInHeap."
+ 	| i freeListsObject |
+ 	i := 0.
+ 	freeListsObject := oldHeap freeListsObject.
+ 	oldHeap allOldSpaceObjectsDo:
+ 		[:obj32|
+ 		(i := i + 1) >= 100000 ifTrue:
+ 			[Transcript nextPut: $:; flush. i := 0].
+ 		 obj32 = freeListsObject
+ 			ifTrue:
+ 				[self cloneFreeLists: obj32]
+ 			ifFalse:
+ 				[(self shouldClone: obj32) ifTrue:
+ 					[self clone: obj32]]]!

Item was added:
+ ----- Method: Spur32BitPreen>>fillInBitsObject:from: (in category 'bootstrap image') -----
+ fillInBitsObject: obj64 from: obj32
+ 	0 to: (oldHeap numBytesOf: obj32) - 1 do:
+ 		[:i|
+ 		newHeap
+ 			storeByte: i
+ 			ofObject: obj64
+ 			withValue: (oldHeap fetchByte: i ofObject: obj32)]!

Item was added:
+ ----- Method: Spur32BitPreen>>fillInClassTable (in category 'bootstrap image') -----
+ fillInClassTable
+ 	self fillInPointerObject: (map at: oldHeap classTableRootObj) from: oldHeap classTableRootObj!

Item was added:
+ ----- Method: Spur32BitPreen>>fillInCompiledMethod:from: (in category 'bootstrap image') -----
+ fillInCompiledMethod: newObj from: oldObj
+ 	"oldInterpreter printOop: oldObj"
+ 	"newInterpreter printOop: newObj"
+ 	0 to: (oldHeap numPointerSlotsOf: oldObj) - 1 do:
+ 		[:i| | oldOop newOop |
+ 		 oldOop := oldHeap fetchPointer: i ofObject: oldObj.
+ 		 newOop := self map32BitOop: oldOop.
+ 		 newHeap
+ 			storePointerUnchecked: i
+ 			ofObject: newObj
+ 			withValue: newOop].
+ 	(oldInterpreter startPCOfMethod: oldObj)
+ 		to: (oldHeap numBytesOf: oldObj) - 1
+ 		do: [:j|
+ 			newHeap
+ 				storeByte: j 
+ 				ofObject: newObj
+ 				withValue: (oldHeap fetchByte: j ofObject: oldObj)]!

Item was added:
+ ----- Method: Spur32BitPreen>>fillInHeap (in category 'bootstrap image') -----
+ fillInHeap
+ 	| heapEnd freeListsObj |
+ 	heapEnd := newHeap freeStart.
+ 	newHeap
+ 		nilObject: (map at: oldHeap nilObject);
+ 		falseObject: (map at: oldHeap falseObject);
+ 		trueObject: (map at: oldHeap trueObject);
+ 		specialObjectsOop: (map at: oldHeap specialObjectsOop);
+ 		lastHash: oldHeap lastHash;
+ 		setHiddenRootsObj: (map at: oldHeap classTableRootObj).
+ 	newHeap segmentManager
+ 		initSegmentForInImageCompilationFrom: newHeap nilObject
+ 		to: heapEnd + newHeap bridgeSize.
+ 	freeListsObj := newHeap objectAfter: newHeap trueObject.
+ 	"Nil-out the free lists."
+ 	newHeap
+ 		fillObj: freeListsObj numSlots: (newHeap numSlotsOf: freeListsObj) with: 0;
+ 		initializeFreeSpacePostLoad: freeListsObj;
+ 		initializePostBootstrap;
+ 		setEndOfMemory: (newHeap segmentManager bridgeAt: 0) + newHeap baseHeaderSize!

Item was added:
+ ----- Method: Spur32BitPreen>>fillInObjects (in category 'bootstrap image') -----
+ fillInObjects
+ 	"interpreter32 printOop: oldObj"
+ 	| i |
+ 	{oldHeap markStack. oldHeap weaklingStack. oldHeap mournQueue} do:
+ 		[:obj|
+ 		obj ~= oldHeap nilObject ifTrue:
+ 			[map at: obj put: (map at: oldHeap nilObject)]].
+ 	i := 0.
+ 	oldHeap allObjectsDo:
+ 		[:oldObj|
+ 		(i := i + 1) >= 10000 ifTrue:
+ 			[Transcript nextPut: $.; flush. i := 0].
+ 		(map at: oldObj ifAbsent: nil)
+ 			ifNotNil:
+ 				[:newObj| | format |
+ 				(newHeap numSlotsOf: newObj) > 0 ifTrue: "filter-out filtered objStack pages"
+ 					[format := oldHeap formatOf: oldObj.
+ 					 (newHeap isPointersFormat: format)
+ 						ifTrue:
+ 							[self fillInPointerObject: newObj from: oldObj]
+ 						ifFalse:
+ 							[(newHeap isCompiledMethodFormat: format)
+ 								ifTrue: [self fillInCompiledMethod: newObj from: oldObj]
+ 								ifFalse: [self fillInBitsObject: newObj from: oldObj]]]]
+ 			ifNil: [self assert: (self isUnmappedObject: oldObj)]]!

Item was added:
+ ----- Method: Spur32BitPreen>>fillInPointerObject:from: (in category 'bootstrap image') -----
+ fillInPointerObject: newObj from: oldObj 
+ 	0 to: (oldHeap numSlotsOf: oldObj) - 1 do:
+ 		[:i|
+ 		 newHeap
+ 			storePointerUnchecked: i
+ 			ofObject: newObj
+ 			withValue: (self map32BitOop: (oldHeap fetchPointer: i ofObject: oldObj))]!

Item was added:
+ ----- Method: Spur32BitPreen>>incrementPCField:ofObject:for: (in category 'bootstrap image') -----
+ incrementPCField: fieldIndex ofObject: obj64 for: method32
+ 	| value nLits |
+ 	value := newHeap fetchPointer: fieldIndex ofObject: obj64.
+ 	(newHeap isIntegerObject: value)
+ 		ifTrue:
+ 			[nLits := oldHeap literalCountOf: method32.
+ 			 newHeap
+ 				storePointerUnchecked: fieldIndex
+ 				ofObject: obj64
+ 				withValue: (newHeap integerObjectOf: nLits + LiteralStart * 4 + (newHeap integerValueOf: value))]
+ 		ifFalse:
+ 			[self assert: (reverseMap at: value) = oldHeap nilObject]!

Item was added:
+ ----- Method: Spur32BitPreen>>initMaps (in category 'initialize-release') -----
+ initMaps
+ 	map := Dictionary new: oldHeap memory size // 32.
+ 	reverseMap := Dictionary new: oldHeap memory size // 32!

Item was added:
+ ----- Method: Spur32BitPreen>>isUnmappedObject: (in category 'bootstrap image') -----
+ isUnmappedObject: obj32
+ 	"Answer if obj32 is an object that is not cloned by the bootstrap."
+ 	^obj32 = oldHeap freeListsObject
+ 	  or: [(oldHeap isValidObjStackPage: obj32)]!

Item was added:
+ ----- Method: Spur32BitPreen>>map32BitOop: (in category 'bootstrap image') -----
+ map32BitOop: obj32
+ 	"oldInterpreter printOop: obj32"
+ 	^map
+ 		at: obj32
+ 		ifAbsent:
+ 			[(oldHeap isImmediateCharacter: obj32)
+ 				ifTrue: [newHeap characterObjectOf: (oldHeap characterValueOf: obj32)]
+ 				ifFalse: [newHeap integerObjectOf: (oldHeap integerValueOf: obj32)]]!

Item was added:
+ ----- Method: Spur32BitPreen>>on: (in category 'public access') -----
+ on: imageName
+ 	(oldInterpreter := StackInterpreterSimulator newWithOptions: #(ObjectMemory Spur32BitMemoryManager))
+ 		openOn: imageName extraMemory: 0.
+ 	oldHeap := oldInterpreter objectMemory.
+ 	imageHeaderFlags := oldInterpreter getImageHeaderFlags.
+ 	savedWindowSize := oldInterpreter savedWindowSize.
+ 	newInterpreter := StackInterpreterSimulator newWithOptions: #(ObjectMemory Spur32BitMemoryManager).
+ 	newHeap := newInterpreter objectMemory.
+ 	newHeap
+ 		allocateMemoryOfSize: oldHeap oldSpaceSize
+ 		newSpaceSize: 8 * 1024 * 1024
+ 		stackSize: 16 * 1024
+ 		codeSize: 0.
+ 	newHeap bootstrapping: true.
+ 	self initMaps!

Item was added:
+ ----- Method: Spur32BitPreen>>preenImage (in category 'public access') -----
+ preenImage
+ 	self cloneObjects.
+ 	self fillInObjects.
+ 	self fillInClassTable.
+ 	self fillInHeap.
+ 	self reportSizes!

Item was added:
+ ----- Method: Spur32BitPreen>>preenImage: (in category 'public access') -----
+ preenImage: imageName
+ 	(Smalltalk classNamed: #FileReference) ifNotNil:
+ 		[^self preenImageUsingFileReference: imageName].
+ 	(Smalltalk classNamed: #FileDirectory) ifNotNil:
+ 		[^self preenImageUsingFileDirectory: imageName].
+ 	self error: 'at a loss as to what file system support to use'!

Item was added:
+ ----- Method: Spur32BitPreen>>preenImageUsingFileDirectory: (in category 'public access') -----
+ preenImageUsingFileDirectory: imageName
+ 	| dirName baseName dir |
+ 	dirName := FileDirectory dirPathFor: imageName.
+ 	baseName := (imageName endsWith: '.image')
+ 					ifTrue: [FileDirectory baseNameFor: imageName]
+ 					ifFalse: [FileDirectory localNameFor: imageName].
+ 	dir := dirName isEmpty ifTrue: [FileDirectory default] ifFalse: [FileDirectory default on: dirName].
+ 	self on: (dir fullNameFor: baseName, '.image').
+ 	[self preenImage]
+ 		on: Halt
+ 		do: [:ex|
+ 			"suppress halts from the usual suspects (development time halts)"
+ 			(#(fullGC compactImage) includes: ex signalerContext sender selector)
+ 				ifTrue: [ex resume]
+ 				ifFalse: [ex pass]].
+ 	self writeSnapshot: (dir fullNameFor: baseName, '-preen.image')
+ 		headerFlags: imageHeaderFlags
+ 		screenSize: savedWindowSize.
+ 	dir deleteFileNamed: baseName, '-preen.changes';
+ 		copyFileNamed: baseName, '.changes' toFileNamed: baseName, '-preen.changes'!

Item was added:
+ ----- Method: Spur32BitPreen>>reportSizes (in category 'bootstrap image') -----
+ reportSizes
+ 	| change oldHeapSize newHeapSize |
+ 	oldHeapSize := oldHeap oldSpaceSize.
+ 	newHeapSize := newHeap oldSpaceSize.
+ 	change := newHeapSize - oldHeapSize / oldHeapSize.
+ 	Transcript
+ 		nextPutAll: 'done.'; cr;
+ 		nextPutAll: 'old heap size: '; nextPutAll: oldHeapSize asStringWithCommas; tab;
+ 		nextPutAll: 'initial new heap size: '; nextPutAll: newHeapSize asStringWithCommas; cr;
+ 		nextPutAll: 'change: '; print: change * 100.0 maxDecimalPlaces: 2; nextPut: $%; cr;
+ 		flush.
+ 	newHeapSize := newHeap endOfMemory
+ 					- newHeap scavenger eden limit
+ 					- newHeap totalFreeListBytes.
+ 	change := newHeapSize - oldHeapSize / oldHeapSize.
+ 	Transcript
+ 		nextPutAll: 'final new heap size: '; nextPutAll: newHeapSize asStringWithCommas; tab;
+ 		nextPutAll: 'change: '; print: change * 100.0 maxDecimalPlaces: 2; nextPut: $%; cr;
+ 		flush!

Item was added:
+ ----- Method: Spur32BitPreen>>shouldClone: (in category 'bootstrap image') -----
+ shouldClone: obj32
+ 	^(oldHeap isValidObjStackPage: obj32) not!

Item was added:
+ ----- Method: Spur32BitPreen>>writeSnapshot:headerFlags:screenSize: (in category 'snapshot') -----
+ writeSnapshot: imageFileName headerFlags: headerFlags screenSize: screenSizeInteger
+ 	newHeap
+ 		checkFreeSpace;
+ 		runLeakCheckerForFullGC.
+ 	newInterpreter
+ 		setImageHeaderFlagsFrom: headerFlags;
+ 		setDisplayForm: nil;
+ 		setSavedWindowSize: savedWindowSize;
+ 		imageName: imageFileName;
+ 		writeImageFileIO.
+ 	Transcript cr; show: 'Done!!'!

Item was changed:
  ----- Method: Spur32to64BitBootstrap>>cloneFreeLists: (in category 'bootstrap image') -----
  cloneFreeLists: obj32
  	| obj64 |
  	obj64 := heap64
  				allocateSlots: heap64 numFreeLists
+ 				format: heap64 wordIndexableFormat
+ 				classIndex: heap64 wordSizeClassIndexPun.
- 				format: heap64 sixtyFourBitIndexableFormat
- 				classIndex: heap64 sixtyFourBitLongsClassIndexPun.
  	reverseMap at: obj64 put: obj32.
  	^map at: obj32 put: obj64!



More information about the Vm-dev mailing list