[Vm-dev] VM Maker: Cog-eem.92.mcz

Thu Sep 19 20:22:35 UTC 2013

Eliot Miranda uploaded a new version of Cog to project VM Maker:
http://source.squeak.org/VMMaker/Cog-eem.92.mcz

==================== Summary ====================

Name: Cog-eem.92
Author: eem
Time: 19 September 2013, 1:22:23.662 pm
UUID: 03144fa4-dcb6-46ce-9d5b-05ed9cdbd3b8
Ancestors: Cog-eem.91

SpurBootstrap:
Rename symbolMap to literalMap and abstract away from direct
access to Smalltalk object's internals, instead simulating bindingOf:,
hence allowing bootstrap of Squeak trunk image that uses Environments.

=============== Diff against Cog-eem.91 ===============

Item was changed:
  Object subclass: #SpurBootstrap
+ 	instanceVariableNames: 'oldHeap newHeap map reverseMap classToIndex oldInterpreter lastClassTablePage literalMap methodClasses installedPrototypes sizeSym rehashSym'
- 	instanceVariableNames: 'oldHeap newHeap map reverseMap classToIndex oldInterpreter lastClassTablePage symbolMap methodClasses installedPrototypes sizeSym rehashSym'
  	classVariableNames: 'NewInterpreter TransformedImage'
  	poolDictionaries: 'VMObjectIndices'
  	category: 'Cog-Bootstrapping'!

  !SpurBootstrap commentStamp: 'eem 9/11/2013 05:45' prior: 0!
  SpurBootstrap bootstraps an image in SpurMemoryManager format from a Squeak V3 + closures format.

  e.g.
  	(SpurBootstrap32 new on: '/Users/eliot/Cog/startreader.image')
  		transform;
  		launch

  Bootstrap issues:
  - should it implement a deterministic Symbol identityHash? This means set a Symbol's identityHash at instance creation time
    based on its string hash so that e.g. MethodDIctionary instances have a deterministic order and don't need to be rehashed on load.
  - should it collapse ContextPart and MethodContext down onto Context (and perhaps eliminate BlockContext)?

  Instance Variables
  	classToIndex:			<Dictionary>
  	lastClassTablePage:	<Integer>
  	map:					<Dictionary>
  	methodClasses:		<Set>
  	newHeap:				<SpurMemoryManager>
  	oldHeap:				<NewObjectMemory>
  	oldInterpreter:			<StackInterpreterSimulator>
  	reverseMap:			<Dictionary>
  	symbolMap:				<Dictionary>

  classToIndex
  	- oldClass to new classIndex map

  lastClassTablePage
  	- oop in newHeap of last classTable page.  U<sed in validation to filter-out class table.

  methodClasses
  	- cache of methodClassAssociations for classes in which modified methods are installed

  map
  	- oldObject to newObject map

  newHeap
  	- the output, bootstrapped image

  oldHeap
  	- the input, image

  oldInterpreter
  	- the interpreter associated with oldHeap, needed for a hack to grab WeakArray

  reverseMap
  	- newObject to oldObject map

  symbolMap
  	- symbol toi symbol oop in oldHeap, used to map prototype methdos to methods in oldHeap!

Item was changed:
  ----- Method: SpurBootstrap>>addNewMethods (in category 'bootstrap methods') -----
  addNewMethods
  	"Get the simulator to add any and all missing methods immediately."
  	| cmaiaSym basSym |
  	cmaiaSym := self findSymbol: #compiledMethodAt:ifAbsent:.
  	basSym := self findSymbol: #basicAddSelector:withMethod:.
  	basSym ifNil:
  		[basSym := self findSymbol: #addSelectorSilently:withMethod:].
  	self allPrototypeClassNamesDo:
  		[:sym :symIsMeta| | class |
+ 		class := self findClassNamed: (literalMap at: sym).
- 		class := self findClassNamed: (symbolMap at: sym).
  		symIsMeta ifTrue: [class := oldHeap fetchClassOfNonImm: class].
  		self prototypeClassNameMetaSelectorMethodDo:
  			[:className :isMeta :selector :method| | methodOrNil |
  			(className = sym
  			 and: [symIsMeta = isMeta]) ifTrue:
  				["probe method dictionary of the class for each method, installing a dummy if not found."
  				 "Transcript cr; nextPutAll: 'checking for '; nextPutAll: selector; flush."
  				 methodOrNil := self interpreter: oldInterpreter
  									object: class
  									perform: cmaiaSym
+ 									withArguments: {literalMap at: selector. oldHeap nilObject}.
- 									withArguments: {symbolMap at: selector. oldHeap nilObject}.
  				 methodOrNil = oldHeap nilObject
  					ifTrue: "no method.  install the real thing now"
  						[Transcript
  							cr;
  							nextPutAll: 'installing ';
  							nextPutAll: className;
  							nextPutAll: (isMeta ifTrue: [' class>>'] ifFalse: ['>>']);
  							store: selector;
  							flush.
  						 self interpreter: oldInterpreter
  							object: class
  							perform: basSym
+ 							withArguments: { literalMap at: selector.
- 							withArguments: { symbolMap at: selector.
  											   self installableMethodFor: method
  												selector: selector
  												className: className
  												isMeta: isMeta}.
  						installedPrototypes add: method selector]
  					ifFalse: "existing method; collect the methodClassAssociation; its needed later"
  						[methodClasses add: (oldInterpreter methodClassAssociationOf: methodOrNil)]]]]!

Item was changed:
  ----- Method: SpurBootstrap>>defineKnownClassIndices (in category 'class indices') -----
  defineKnownClassIndices
  	"The classTable is laid out
  		- to make it easy to map immediates to classes; the tag pattern of an immediate is its class index.
  		  hence there are two entries for SmallInteger
  		- to assign small indices to well-known classes such as Array, Message et al
  		- to leave plenty of room for new known classes; hence the first page contains only well-known classes
  		- to enable overlaps and avoid conflicts with indices in the specialObjectsArray (?really? eem)
  		- to provide a WeakArray pun for the pages of the table itself so that these do not show up as instances of WeakArray"
  	| classMethodContext classBlockClosure classMessage "no api method for these" |
  	classMessage := oldHeap splObj: (VMObjectIndices bindingOf: #ClassMessage) value.
  	classMethodContext := oldHeap splObj: (VMObjectIndices bindingOf: #ClassMethodContext) value.
  	classBlockClosure := oldHeap splObj: (VMObjectIndices bindingOf: #ClassBlockClosure) value.
  	"c.f. SpurMemoryManager class>>intializeCompactClassIndices".
  	classToIndex keysDo:
  		[:oldClass|
  		self assert: (oldInterpreter addressCouldBeClassObj: oldClass)].
  	classToIndex
  		at: oldHeap classSmallInteger put: 1; "N.B. must fill-in index 3 manually"
  		at: oldHeap classCharacter put: 2;
  		"at: oldHeap classSmallInteger put: 3" "N.B. must fill-in index 3 manually"
  		"leave room for up to 15 tagged classes"
  		"leave room for up to 16 puns"
  		at: oldHeap classLargeNegativeInteger put: 32;
  		at: oldHeap classLargePositiveInteger put: 33;
  		at: oldHeap classFloat put: 34;

  		at: "oldHeap" classMessage put: 35;
  		at: "oldHeap" classMethodContext put: 36;
  		at: "oldHeap" classBlockClosure put: 37;

  		at: oldHeap classSemaphore put: 48;
+ 		"at: oldHeap classMutex put: 49; see below"
- 		at: oldHeap classMutex put: 49;

  		at: oldHeap classByteArray put: 50;
  		at: oldHeap classArray put: 51;
  		at: oldHeap classString put: 52;
  		at: oldHeap classBitmap put: 53;
+ 		at: oldHeap classPoint put: 54.
- 		at: oldHeap classPoint put: 54;

+ 	{{oldHeap classMutex. 49}.
+ 	 {oldHeap classExternalAddress. 128}.
+ 	 {oldHeap classExternalData. 129}.
+ 	 {oldHeap classExternalFunction. 130}.
+ 	 {oldHeap classExternalLibrary. 131}.
+ 	 {oldHeap classExternalStructure. 132}.
+ 	 {oldHeap classAlien. 133}.
+ 	 {oldHeap classUnsafeAlien. 134}}
+ 		do: [:pair|
+ 			[:oop :index|
+ 			oop ~= oldHeap nilObject ifTrue:
+ 				[classToIndex at: oop put: index]] valueWithArguments: pair].
+ 
- 		at: oldHeap classExternalAddress put: 128;
- 		at: oldHeap classExternalData put: 129;
- 		at: oldHeap classExternalFunction put: 130;
- 		at: oldHeap classExternalLibrary put: 131;
- 		at: oldHeap classExternalStructure put: 132;
- 		at: oldHeap classAlien put: 133;
- 		at: oldHeap classUnsafeAlien put: 134.
  	classToIndex keysDo:
  		[:oldClass|
  		self assert: (oldInterpreter addressCouldBeClassObj: oldClass)]!

Item was added:
+ ----- Method: SpurBootstrap>>findRequiredGlobals (in category 'bootstrap methods') -----
+ findRequiredGlobals
+ 	"Look for the necessary gobal bindings in the prototype methods in the old image.
+ 	 This has to be done early by sending bindingOf: to Smalltalk."
+ 	| globals bindingOf |
+ 	globals := Set new.
+ 	self prototypeClassNameMetaSelectorMethodDo:
+ 		[:c :m :s :method|
+ 		globals addAll: (method literals select: [:l|
+ 										l isVariableBinding
+ 										and: [l key isSymbol
+ 										and: [(Smalltalk bindingOf: l key) == l]]])].
+ 	bindingOf := self findSymbol: #bindingOf:.
+ 	self withExecutableInterpreter: oldInterpreter
+ 		do:	[globals do:
+ 				[:global|
+ 				literalMap
+ 					at: global
+ 					put: (self interpreter: oldInterpreter
+ 							object: (oldHeap splObj: 8) "Smalltalk"
+ 							perform: bindingOf
+ 							withArguments: {self findSymbol: global key})]]!

Item was changed:
  ----- Method: SpurBootstrap>>installModifiedMethods (in category 'bootstrap methods') -----
  installModifiedMethods
  	"Install all the methods in the class-side method prototypes protocol in the relevant classes
  	 in the new image.  First use the simulator to get the image to intern all symbols and add
  	 dummy methods under new selectors.  With that done we can manually replace the relevant
  	 methods with the prototypes, mapping selectors and global variables as required."
- 	symbolMap := Dictionary new.
- 	methodClasses := Set new.
- 	installedPrototypes := Set new.
  	self withExecutableInterpreter: oldInterpreter
  		do: [self internAllSymbols.
  			 self addNewMethods.
  			 self replaceMethods.
  			 self modifyCharacterMethods]!

Item was changed:
  ----- Method: SpurBootstrap>>installableMethodFor:selector:className:isMeta: (in category 'bootstrap methods') -----
  installableMethodFor: aCompiledMethod selector: selector className: className isMeta: isMeta
  	| compiledMethodClass sourcelessMethod bytes newMethod methodClass |
  	compiledMethodClass := self findClassNamed: (self findSymbol: #CompiledMethod).
  	"the prototypes hve source pointers.  the Character methods to be replaced don't."
  	sourcelessMethod := aCompiledMethod trailer hasSourcePointer
  							ifTrue: [aCompiledMethod copyWithTempsFromMethodNode: aCompiledMethod methodNode]
  							ifFalse: [aCompiledMethod].
  	bytes := sourcelessMethod size - sourcelessMethod initialPC + 1.
  	newMethod := self
  					interpreter: oldInterpreter
  					object: compiledMethodClass
  					perform: (self findSymbol: #newMethod:header:)
  					withArguments: { oldHeap integerObjectOf: bytes.
  									   oldHeap integerObjectOf: sourcelessMethod header }.
  	1 to: sourcelessMethod numLiterals - 1 do:
  		[:i| | literal oop |
  		literal := sourcelessMethod literalAt: i.
  		literal isMethodProperties ifTrue:
  			[literal := selector].
  		oop := (literal isLiteral or: [literal isVariableBinding])
  					ifTrue:
  						[literal isInteger
  							ifTrue: [oldHeap integerObjectOf: literal]
+ 							ifFalse: [literalMap at: literal ifAbsent: [self findLiteral: literal]]]
- 							ifFalse: [symbolMap at: literal ifAbsent: [self findLiteral: literal]]]
  					ifFalse: "should be a VMObjectProxy"
  						[literal oop].
  		oldHeap storePointer: i ofObject: newMethod withValue: oop].
  	methodClass := self methodClassForClassName: className isMeta: isMeta.
  	oldHeap storePointer: sourcelessMethod numLiterals ofObject: newMethod withValue: methodClass.
  	sourcelessMethod initialPC to: sourcelessMethod size do:
  		[:i|
  		oldHeap storeByte: i - 1 ofObject: newMethod withValue: (sourcelessMethod byteAt: i)].
  	^newMethod!

Item was changed:
  ----- Method: SpurBootstrap>>internAllSymbols (in category 'bootstrap methods') -----
  internAllSymbols
  	"Ensure that all symbols in the method prototypes are interned so that later we can install them.
  	 Enter them into the map, this system's symbol -> oldHeap's version.
  	 Do this by interpreting Symbol intern: 'aSymbol' for each symbol."
  	| internSym |
  	internSym := self findSymbol: #intern:.
  	self allPrototypeMethodSymbols do:
  		[:sym|
  		(self findSymbol: sym)
+ 			ifNotNil: [:imageSym| literalMap at: sym put: imageSym]
- 			ifNotNil: [:imageSym| symbolMap at: sym put: imageSym]
  			ifNil:[Transcript cr; nextPutAll: 'interning '; nextPutAll: sym; flush.
  				"Interpret Symbol intern: sym to ... intern it :-)"
+ 				literalMap
- 				symbolMap
  					at: sym
  					put: (self interpreter: oldInterpreter
  							object: self symbolClass
  							perform: internSym
  							withArguments: {self stringFor: sym})]].
+ 	literalMap keysAndValuesDo:
+ 		[:symOrGlobal :imageSymOrGlobal|
+ 		symOrGlobal isSymbol ifTrue:
+ 			[self assert: symOrGlobal = (oldHeap stringOf: imageSymOrGlobal)]]!
- 	symbolMap keysAndValuesDo:
- 		[:sym :imageSym|
- 		self assert: sym = (oldHeap stringOf: imageSym)]!

Item was changed:
  ----- Method: SpurBootstrap>>methodClassForClassName:isMeta: (in category 'bootstrap methods') -----
  methodClassForClassName: classNameSymbol isMeta: isMeta 
  	| class |
+ 	class := self findClassNamed: (literalMap at: classNameSymbol).
- 	class := self findClassNamed: (symbolMap at: classNameSymbol).
  	isMeta ifTrue: [class := oldHeap fetchClassOfNonImm: class].
  	methodClasses do:
  		[:mca|
  		class = (oldHeap fetchPointer: ValueIndex ofObject: mca) ifTrue:
  			[^mca]].
  	oldHeap allObjectsDo:
  		[:o| | methodClassAssociation |
  		(oldHeap isCompiledMethod: o) ifTrue:
  			[methodClassAssociation := oldInterpreter methodClassAssociationOf: o.
  			 class == (oldHeap fetchPointer: ValueIndex ofObject: methodClassAssociation) ifTrue:
  				[methodClasses add: methodClassAssociation.
  				 ^methodClassAssociation]]].
  	self error: 'could not find methodClassAssociation for ', classNameSymbol, (isMeta ifTrue: [' class'] ifFalse:[''])!

Item was changed:
  ----- Method: SpurBootstrap>>on: (in category 'initialize-release') -----
  on: imageName
  	StackInterpreter initializeWithOptions: Dictionary new.
  	oldInterpreter := StackInterpreterSimulator new.
  	oldInterpreter openOn: imageName extraMemory: 0.
  	oldHeap := oldInterpreter objectMemory.
  	newHeap := Spur32BitMMLESimulator new.
  	newHeap
  		allocateMemoryOfSize: (oldHeap youngStart * 5 / 4 roundUpTo: 8)
  		newSpaceSize: 1024 * 1024
  		codeSize: 1024 * 1024.
  		newHeap setCheckForLeaks: 15 - 4. "don't check become"
  	map := Dictionary new: oldHeap memory size // 4.
  	reverseMap := Dictionary new: oldHeap memory size // 4.
+ 	classToIndex := Dictionary new: 1024.
+ 	literalMap := IdentityDictionary new.
+ 	methodClasses := Set new.
+ 	installedPrototypes := Set new!
- 	classToIndex := Dictionary new: 1024!

Item was changed:
  ----- Method: SpurBootstrap>>replaceMethods (in category 'bootstrap methods') -----
  replaceMethods
  	"Replace all the modified method prototypes."
  	self allPrototypeClassNamesDo:
  		[:sym :symIsMeta| | class |
+ 		class := self findClassNamed: (literalMap at: sym).
- 		class := self findClassNamed: (symbolMap at: sym).
  		symIsMeta ifTrue: [class := oldHeap fetchClassOfNonImm: class].
  		self prototypeClassNameMetaSelectorMethodDo:
  			[:className :isMeta :selector :method| | replacement methodDict index |
  			(className = sym
  			 and: [symIsMeta = isMeta]) ifTrue:
  				[(installedPrototypes includes: method selector) ifFalse:
  					["probe method dictionary of the class for each method, installing a dummy if not found."
  					Transcript
  						cr;
  						nextPutAll: 'replacing ';
  						nextPutAll: className;
  						nextPutAll: (isMeta ifTrue: [' class>>'] ifFalse: ['>>']);
  						store: selector;
  						flush.
  					replacement := self installableMethodFor: method
  										selector: selector
  										className: className
  										isMeta: isMeta.
  					methodDict := oldHeap fetchPointer: MethodDictionaryIndex ofObject: class.
+ 					index := self indexOfSelector: (literalMap at: selector) in: methodDict.
- 					index := self indexOfSelector: (symbolMap at: selector) in: methodDict.
  					oldHeap
  						storePointer: index - SelectorStart
  						ofObject: (oldHeap fetchPointer: MethodArrayIndex ofObject: methodDict)
  						withValue: replacement.
  					installedPrototypes add: method selector]]]]!

Item was changed:
  ----- Method: SpurBootstrap>>replacementForCharacterMethod: (in category 'bootstrap methods') -----
  replacementForCharacterMethod: characterMethodOop
  	"Answer a replacement method for the argument if it refers
  	 to Character's old inst var value.  Otherwise answer nil."
  	| proxy asIntegerProxy clone assembly newInsts newMethod |
  	"(oldHeap stringOf: (oldHeap longAt: characterMethodOop + (oldHeap lastPointerOf: characterMethodOop) - 4)) = 'isOctetCharacter' ifTrue:
  		[self halt]."
  	proxy := VMCompiledMethodProxy new
  				for: characterMethodOop
  				coInterpreter: oldInterpreter
  				objectMemory: oldHeap.
  	clone := self cloneMethodProxy: proxy.
  	"Quick methods accessing value should have been replaced."
  	clone isReturnField ifTrue: [self halt].
  	clone hasInstVarRef ifFalse:
  		[^nil].
  	clone setSourcePointer: 0.
  	asIntegerProxy := VMObjectProxy new
+ 							for: (literalMap at: #asInteger)
- 							for: (symbolMap at: #asInteger)
  							coInterpreter: oldInterpreter
  							objectMemory: oldHeap.
  	assembly := BytecodeDisassembler new disassemble: clone.
  	assembly literals: (assembly literals allButLast: 2), {asIntegerProxy}, (assembly literals last: 2).
  		"Do this by looking for index of pushReceiverVariable: and replacing it by pushSelf, send asInteger"
  	newInsts := (assembly instructions piecesCutWhere:
  					[:msgOrLabelAssoc :nextInst|
  					 msgOrLabelAssoc isVariableBinding not
  					 and: [msgOrLabelAssoc selector == #pushReceiverVariable:]]) fold:
  				[:a :b|
  				 a allButLast,
  				 {	Message selector: #pushReceiver.
  					Message
  						selector: #send:super:numArgs:
  						arguments: {asIntegerProxy. false. 0}},
  				 b].
  	assembly instructions: newInsts.
  	newMethod := assembly assemble.
  	^self
  		installableMethodFor: newMethod
  		selector: clone selector
  		className: #Character
  		isMeta: false!

Item was changed:
  ----- Method: SpurBootstrap>>transform (in category 'bootstrap image') -----
  transform
  	self rememberRehashSymbol.
+ 	self findRequiredGlobals.
  	self installModifiedMethods.
  	self bootstrapImage.
  	self validate.
  	self rehashImage.
  	self followForwardingPointers.
  	self scavengeImage.
  	Transcript nextPutAll: 'done.'; flush.!