[Vm-dev] mmap, changes to co-ordinate mapping to particular address to avoid oops address remapping at startup time.

[David T. Lewis]

On Fri, Oct 10, 2008 at 09:36:51PM -0700, John M McIntosh wrote:
> 
> In the past year or so there has been discussion of using mmap to set  
> a start address for the offset of object memory so that memory address  
> swizzling does not occur.
> Also various operating systems vendors say using mmap to page in large  
> files is faster than using fread, this in fact is true and noticeable  
> on the iPhone.
> 
> This requires a change to VMMaker so that sqAllocateMemory can pass  
> the file pointer and the header size, since in the
> sqAllocateMemory we need to figure out the file (image size) and also  
> understand the headersize of the image file to return a proper address  
> to work with.

John,

If I understand correctly, the change in #readImageFromFile:HeapSize:StartingAt:
might look like this:

  "allocate a contiguous block of memory for the Squeak heap"
  memory := self
      allocateMemory: heapSize
      minimum: minimumMemory
      imageFile: f
      headerSize: headerSize.

For backward compatibility in the support code (e.g. for the existing unix
platform code), a define could be provided like this:

  #define allocateMemoryminimumimageFileheaderSize(heapSize, minimumMemory, f, headerSize) \
      sqAllocateMemory(minimumMemory, heapSize)

And for the iPhone VM, you could have a macro like this:

  #define allocateMemoryminimumimageFileheaderSize(heapSize, minimumMemory, f, headerSize) \
      sqAllocateMemoryMac(minimumMemory, heapSize, f, headerSize) {

Does that sound right? Change set attached for the VMMaker part.

- Dave

-------------- next part --------------
'From Squeak3.8.2a of ''26 Oct 2007'' [latest update: #6748] on 23 October 2008 at 8:43:58 pm'!
"Change Set:		Interpreter-readImageFromFile-jmm-dtl
Date:			23 October 2008
Author:			David T. Lewis

Pass image file and image header length to object memory allocation function in order to enable mmap loading without address swizzling, per jmm proposal.

For backward compatibility in support code:
#define allocateMemoryminimumimageFileheaderSize(heapSize, minimumMemory, f, headerSize) sqAllocateMemory(minimumMemory, heapSize)
"!


!Interpreter methodsFor: 'image save/restore' stamp: 'dtl 10/23/2008 20:32'!
readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset
	"Read an image from the given file stream, allocating the given amount of memory to its object heap. Fail if the image has an unknown format or requires more than the given amount of memory."
	"Details: This method detects when the image was stored on a machine with the opposite byte ordering from this machine and swaps the bytes automatically. Furthermore, it allows the header information to start 512 bytes into the file, since some file transfer programs for the Macintosh apparently prepend a Mac-specific header of this size. Note that this same 512 bytes of prefix area could also be used to store an exec command on Unix systems, allowing one to launch Smalltalk by invoking the image name as a command."
	"This code is based on C code by Ian Piumarta and Smalltalk code by Tim Rowledge. Many thanks to both of you!!!!"

	| swapBytes headerStart headerSize dataSize oldBaseAddr minimumMemory memStart bytesRead bytesToShift heapSize |
	self var: #f type: 'sqImageFile '.
	self var: #headerStart type: 'squeakFileOffsetType '.
	self var: #dataSize type: 'size_t '.
	self var: #imageOffset type: 'squeakFileOffsetType '.

	swapBytes := self checkImageVersionFrom: f startingAt: imageOffset.
	headerStart := (self sqImageFilePosition: f) - BytesPerWord.  "record header start position"

	headerSize			:= self getLongFromFile: f swap: swapBytes.
	dataSize				:= self getLongFromFile: f swap: swapBytes.
	oldBaseAddr			:= self getLongFromFile: f swap: swapBytes.
	specialObjectsOop	:= self getLongFromFile: f swap: swapBytes.
	lastHash			:= self getLongFromFile: f swap: swapBytes.
	savedWindowSize	:= self getLongFromFile: f swap: swapBytes.
	fullScreenFlag		:= self getLongFromFile: f swap: swapBytes.
	extraVMMemory		:= self getLongFromFile: f swap: swapBytes.

	lastHash = 0 ifTrue: [
		"lastHash wasn't stored (e.g. by the cloner); use 999 as the seed"
		lastHash := 999].

	"decrease Squeak object heap to leave extra memory for the VM"
	heapSize := self cCode: 'reserveExtraCHeapBytes(desiredHeapSize, extraVMMemory)'.

	"compare memory requirements with availability".
	minimumMemory := dataSize + 100000.  "need at least 100K of breathing room"
	heapSize < minimumMemory ifTrue: [
		self insufficientMemorySpecifiedError].

	"allocate a contiguous block of memory for the Squeak heap"
	memory := self
		allocateMemory: heapSize
		minimum: minimumMemory
		imageFile: f
		headerSize: headerSize.
	memory = nil ifTrue: [self insufficientMemoryAvailableError].

	memStart := self startOfMemory.
	memoryLimit := (memStart + heapSize) - 24.  "decrease memoryLimit a tad for safety"
	endOfMemory := memStart + dataSize.

	"position file after the header"
	self sqImageFile: f Seek: headerStart + headerSize.

	"read in the image in bulk, then swap the bytes if necessary"
	bytesRead := self
		sqImage: (self pointerForOop: memory)
		read: f
		size: (self cCode: 'sizeof(unsigned char)')
		length: dataSize.
	bytesRead ~= dataSize ifTrue: [self unableToReadImageError].

	headerTypeBytes at: 0 put: BytesPerWord * 2.		"3-word header (type 0)"	
	headerTypeBytes at: 1 put: BytesPerWord.		"2-word header (type 1)"
	headerTypeBytes at: 2 put: 0.					"free chunk (type 2)"	
	headerTypeBytes at: 3 put: 0.					"1-word header (type 3)"

	swapBytes ifTrue: [self reverseBytesInImage].

	"compute difference between old and new memory base addresses"
	bytesToShift := memStart - oldBaseAddr.
	self initializeInterpreter: bytesToShift.  "adjusts all oops to new location"
	self isBigEnder. "work out the machine endianness and cache the answer"

	^ dataSize
! !