I am attempting to zero all ooP pointers before writng an image to disk.
I have added the following code to Interpreter-#writeImage:
writeImageFile: imageBytes
... self adjustAllOopsBy: (0 - memory). self writeImageFileIO: imageBytes. self adjustAllOopsBy: memory. ...
I managed to build a unix version of the vm with this change but it did not work as hoped. Starting an image saved with this vm crashes giving a bus error.
any ideas?
I am also unable to find any documentation on generating the .app version of the unix vm for Mac OS X.
thanks in advance
Keith
_
To build this into squeak:
Loaded 3.9.8-12 stuff from http://www.squeakvm.org/unix/release/
I have the current trunk obtained by:
svn co http://squeakvm.org/svn/squeak/trunk squeak
I have
http://www.squeakvm.org/unix/release/unix-3.9-12.image.gz http://www.squeakvm.org/unix/release/unix-3.9-12.changes.gz http://www.squeakvm.org/unix/release/unix-3.9-12.vmm.config
I am on Mac OS X, using John 's OS X VM 3.8.18beta1U
The above config file is not visible by the VMaker Tool 'load' button, when I save my own config, that one is.
Clicking the platform button:
UnixOSProcessPlugIn-#isResponsibleForThisPlatform
calls Smalltalk osVersion rather than SmalltalkImage current.
On Sep 29, 2007, at 21:56 , Keith Hodges wrote:
I am attempting to zero all ooP pointers before writng an image to disk.
I have added the following code to Interpreter-#writeImage:
writeImageFile: imageBytes
... self adjustAllOopsBy: (0 - memory). self writeImageFileIO: imageBytes. self adjustAllOopsBy: memory. ...
I managed to build a unix version of the vm with this change but it did not work as hoped. Starting an image saved with this vm crashes giving a bus error.
any ideas?
The base address is stored in the image, so before calling writeImageFileIO you would have to set memory to zero I guess.
- Bert -
Oops in
Interpreter>>writeImageFileIO:
you need to change the self putLong: (self startOfMemory) toFile: f.
to I think self putLong: 0 toFile: f.
On Sep 29, 2007, at 12:56 PM, Keith Hodges wrote:
I managed to build a unix version of the vm with this change but it did not work as hoped. Starting an image saved with this vm crashes giving a bus error.
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
I was so excited at the prospect of getting a certificate!
Interpreter>>writeImageFileIO:
you need to change the self putLong: (self startOfMemory) toFile: f.
to I think self putLong: 0 toFile: f.
I had tried that:
but still no joy.
I have included my image writing method below
thanks for the support
Keith
----
writeImageFileIO: imageBytes
| headerStart headerSize f bytesWritten sCWIfn okToWrite | self var: #f type: 'sqImageFile'. self var: #headerStart type: 'squeakFileOffsetType '. self var: #sCWIfn type: 'void *'.
"kph mod - to support binary diffs"
self adjustAllOopsBy: (0 - self startOfMemory).
"If the security plugin can be loaded, use it to check for write permission. If not, assume it's ok" sCWIfn := self ioLoadFunction: 'secCanWriteImage' From: 'SecurityPlugin'. sCWIfn ~= 0 ifTrue:[okToWrite := self cCode: '((sqInt (*)(void))sCWIfn)()'. okToWrite ifFalse:[^self primitiveFail]].
"local constants" headerStart := 0. headerSize := 64. "header size in bytes; do not change!"
f := self cCode: 'sqImageFileOpen(imageName, "wb")'. f = nil ifTrue: [ "could not open the image file for writing" self success: false. ^ nil].
headerStart := self cCode: 'sqImageFileStartLocation(f,imageName,headerSize+imageBytes)'. self cCode: '/* Note: on Unix systems one could put an exec command here, padded to 512 bytes */'. "position file to start of header" self sqImageFile: f Seek: headerStart.
self putLong: (self imageFormatVersion) toFile: f. self putLong: headerSize toFile: f. self putLong: imageBytes toFile: f. self putLong: 0 "(self startOfMemory) kph mod - for binary diffs" toFile: f. self putLong: specialObjectsOop toFile: f. self putLong: lastHash toFile: f. self putLong: (self ioScreenSize) toFile: f. self putLong: fullScreenFlag toFile: f. self putLong: extraVMMemory toFile: f. 1 to: 7 do: [:i | self putLong: 0 toFile: f]. "fill remaining header words with zeros" successFlag ifFalse: [ "file write or seek failure" self cCode: 'sqImageFileClose(f)'. ^ nil].
"position file after the header" self sqImageFile: f Seek: headerStart + headerSize.
"write the image data" bytesWritten := self cCode: 'sqImageFileWrite(pointerForOop(memory), sizeof(unsigned char), imageBytes, f)'. self success: bytesWritten = imageBytes. self cCode: 'sqImageFileClose(f)'.
self adjustAllOopsBy: (self startOfMemory).
perhaps my sources are not being regenerated properly.. ill check
Keith
Ok, I needed to adjust the saved value of specialObjectsOop.
it worked!
Now I need to ensure that the byte ordering is consistent
Keith
Now I need to ensure that the byte ordering is consistent
Keith
I am planning to build a minimal squeak vm for unix and windows which can be added to the mercurial installation as an encoder for files of type *.image (the unix vm build will be sufficient to do the job on Mac OS X).
This is the best way that I can think of for supporting squeak activities using vm's and images of every description. This way the normalizing code doesn't need to be in every users vm.
Mercurial needs the encoder to be invokable from the commandline like so, which should be doable.
squeak -normalize INFILE OUTFILE
In this case I dont even need the image to properly start up, so I dont really need to unswizzle things having written the image.
I am not having much sucess with saving the image little-endian
any thoughts would be appreciated
Keith
--- appendix: The Story so far:
Interpreter-putLong: aWord toFile: aFile swap: swapFlag "swap byte order as requested"
| objectsWritten w | self var: #aFile type: 'sqImageFile '.
w := aWord. swapFlag ifTrue: [ w := self byteSwapped: w ].
objectsWritten := self cCode: 'sqImageFileWrite(&w, sizeof(w), 1, aFile)'. self success: objectsWritten = 1.
Interpreter-writeImageFileIO: imageBytes
| headerStart headerSize f bytesWritten sCWIfn okToWrite swapToLittle |
self var: #f type: 'sqImageFile'. self var: #headerStart type: 'squeakFileOffsetType '. self var: #sCWIfn type: 'void *'.
"If the security plugin can be loaded, use it to check for write permission. If not, assume it's ok" sCWIfn := self ioLoadFunction: 'secCanWriteImage' From: 'SecurityPlugin'. sCWIfn ~= 0 ifTrue:[okToWrite := self cCode: '((sqInt (*)(void))sCWIfn)()'. okToWrite ifFalse:[^self primitiveFail]].
"kph mod - to support binary diffs" self adjustAllOopsBy: (0 - self startOfMemory). swapToLittle := self isBigEnder.
"local constants" headerStart := 0. headerSize := 64. "header size in bytes; do not change!"
f := self cCode: 'sqImageFileOpen(imageName, "wb")'. f = nil ifTrue: [ "could not open the image file for writing" self success: false. ^ nil].
headerStart := self cCode: 'sqImageFileStartLocation(f,imageName,headerSize+imageBytes)'. self cCode: '/* Note: on Unix systems one could put an exec command here, padded to 512 bytes */'. "position file to start of header" self sqImageFile: f Seek: headerStart.
self putLong: (self imageFormatVersion) toFile: f swap: swapToLittle. self putLong: headerSize toFile: f swap: swapToLittle. self putLong: imageBytes toFile: f swap: swapToLittle. self putLong: 0 "(self startOfMemory) kph mod - for binary diffs" toFile: f swap: swapToLittle. self putLong: (specialObjectsOop - self startOfMemory) toFile: f swap: swapToLittle. self putLong: lastHash toFile: f swap: swapToLittle. self putLong: (self ioScreenSize) toFile: f swap: swapToLittle. self putLong: fullScreenFlag toFile: f swap: swapToLittle. self putLong: extraVMMemory toFile: f swap: swapToLittle. 1 to: 7 do: [:i | self putLong: 0 toFile: f]. "fill remaining header words with zeros" successFlag ifFalse: [ "file write or seek failure" self cCode: 'sqImageFileClose(f)'. ^ nil].
"position file after the header" self sqImageFile: f Seek: headerStart + headerSize.
"write the image data" swapToLittle ifTrue: [ self reverseBytesInImage].
bytesWritten := self cCode: 'sqImageFileWrite(pointerForOop(memory), sizeof(unsigned char), imageBytes, f)'. self success: bytesWritten = imageBytes. self cCode: 'sqImageFileClose(f)'.
swapToLittle ifTrue: [ self reverseBytesInImage]. self adjustAllOopsBy: (self startOfMemory).
Now I need to ensure that the byte ordering is consistent
Keith
I am planning to build a minimal squeak vm for unix and windows which can be added to the mercurial installation as an encoder for files of type *.image (the unix vm build will be sufficient to do the job on Mac OS X).
This is the best way that I can think of for supporting squeak activities using vm's and images of every description. This way the normalizing code doesn't need to be in every users vm.
Mercurial needs the encoder to be invokable from the commandline like so, which should be doable.
squeak -normalize INFILE OUTFILE
In this case I dont even need the image to properly start up, so I dont really need to unswizzle things having written the image.
I am not having much sucess with saving the image little-endian
any thoughts would be appreciated
Keith
appendix: The Story so far:
------------------------------------------
Interpreter-putLong: aWord toFile: aFile swap: swapFlag "sway byte order as requested"
| objectsWritten w | self var: #aFile type: 'sqImageFile '.
w := aWord. swapFlag ifTrue: [ w := self byteSwapped: aWord ].
objectsWritten := self cCode: 'sqImageFileWrite(&w, sizeof(w), 1, aFile)'. self success: objectsWritten = 1.
------------------------------------------
Interpreter-writeImageFileIO: imageBytes
| headerStart headerSize f bytesWritten sCWIfn okToWrite swapToLittle |
self var: #f type: 'sqImageFile'. self var: #headerStart type: 'squeakFileOffsetType '. self var: #sCWIfn type: 'void *'.
"If the security plugin can be loaded, use it to check for write permission. If not, assume it's ok" sCWIfn := self ioLoadFunction: 'secCanWriteImage' From: 'SecurityPlugin'. sCWIfn ~= 0 ifTrue:[okToWrite := self cCode: '((sqInt (*)(void))sCWIfn)()'. okToWrite ifFalse:[^self primitiveFail]].
"kph mod - to support binary diffs" self adjustAllOopsBy: (0 - self startOfMemory). swapToLittle := self isBigEnder.
"local constants" headerStart := 0. headerSize := 64. "header size in bytes; do not change!"
f := self cCode: 'sqImageFileOpen(imageName, "wb")'. f = nil ifTrue: [ "could not open the image file for writing" self success: false. ^ nil].
headerStart := self cCode: 'sqImageFileStartLocation(f,imageName,headerSize+imageBytes)'. self cCode: '/* Note: on Unix systems one could put an exec command here, padded to 512 bytes */'. "position file to start of header" self sqImageFile: f Seek: headerStart.
self putLong: (self imageFormatVersion) toFile: f swap: swapToLittle. self putLong: headerSize toFile: f swap: swapToLittle. self putLong: imageBytes toFile: f swap: swapToLittle. self putLong: 0 "(self startOfMemory) kph mod - for binary diffs" toFile: f swap: swapToLittle. self putLong: (specialObjectsOop - self startOfMemory) toFile: f swap: swapToLittle. self putLong: lastHash toFile: f swap: swapToLittle. self putLong: (self ioScreenSize) toFile: f swap: swapToLittle. self putLong: fullScreenFlag toFile: f swap: swapToLittle. self putLong: extraVMMemory toFile: f swap: swapToLittle. 1 to: 7 do: [:i | self putLong: 0 toFile: f]. "fill remaining header words with zeros" successFlag ifFalse: [ "file write or seek failure" self cCode: 'sqImageFileClose(f)'. ^ nil].
"position file after the header" self sqImageFile: f Seek: headerStart + headerSize.
"write the image data" swapToLittle ifTrue: [ self reverseBytesInImage].
bytesWritten := self cCode: 'sqImageFileWrite(pointerForOop(memory), sizeof(unsigned char), imageBytes, f)'. self success: bytesWritten = imageBytes. self cCode: 'sqImageFileClose(f)'.
swapToLittle ifTrue: [ self reverseBytesInImage]. self adjustAllOopsBy: (self startOfMemory).
I am surprise to find that subclassing Interpreter to make my own refinements does not work is this to be expected?
Keith
Having touched upon the hallowed ground of VMMaking, I have started to desecrate the altar itself...
I have attempted to add limited but "useful-enough" inheritance to VMMaker.
Is there a repository to which I can upload any changes for review/feedback/integration?
With my tweaks if you implement a subclass of Interpreter, and the class method #inherits ^ true, then you can implement an alternative interpreter by adding or overriding methods. The same should be true for plugin's
[BUG] I discover that the value of the field: "Interpreter class name" does not successfully save/load in the vmm.config files
I have attempted to test this but subclassing LargeIntegerPlugin twice and generating both, and it seemed to work, though I have not fully tested it.
As a trival example use case: The comment in LargeIntegersPlugin-buildCodeGeneratorUpTo: someClass indicates that there is no UI means to enable debugCode generation. Making LargeIntegersPluginDebug a subclass of LargeIntegersPlugin could add the single needed method. Both options would appear in the available plugins enabling the user to select the debug/nondebug versions.
now.. back to normalizing!
Keith
The plan:
Mercurial installations have the following in their hgrc configuration file:
A)
[encode] **.image :tempfile squeaknormalize -normalizeNoStart INFILE OUTFILE
B) [encode] **.image :tempfile squeaknormalize -headless -normalize INFILE OUTFILE /path/to/script.st
Whenever mercurial commits a *.image file to the repository it will pass it through this filter. squeaknormalize being a modified vm.
Option A) reads, writes and quits the image without even starting the interpreter Option B) runs a script, which is then responsible for snapshotting and quitting
The script could perform some simple image tidying such as flushing MC caches etc.
When reading parameters, both options sneakily grab the first parameter (INFILE) into "normalizingInputImageName", the normal document (OUTFILE) is the next parameter and this becomes the current "imageName"
Typically mercurial uses temp files of the form /tmp/hg-filter-123 without an extension to support this I have had to patch out the automatic appending of 'image' to the document parameter.
This scheme described above is implemented but not tested or debugged.
I still haven't succeeded in getting the image to save littleEndian on my mac..
best regards
Keith
Having just watched a video about OLPC and seen how little storage they have. Have any OLPC folks done this normalizing the image already?
Is the OLPC big or little endian?
I may be having a go at testing my latest this weekend, would anoyne be willing to help out with the windows version since I lack tools/machine etc?
best regards
Keith
(a) The OLPC actually uses a compressed file system already, because of that Published Sophie Books for the OLPC use a zipped file with no compression. The zipped file will live in the journal at some point.
(b) Does setting oops values back to start of zero actually make it more compressable, versus say a start value of 0x00004000? Otherwise the computational energy used to traverse the image is technically wasted.
(c) ittle/big endian? OLPC is based on the geod aka intel instruction set.
http://wiki.laptop.org/go/Hardware_specification
On Oct 4, 2007, at 6:53 PM, Keith Hodges wrote:
Having just watched a video about OLPC and seen how little storage they have. Have any OLPC folks done this normalizing the image already?
Is the OLPC big or little endian?
I may be having a go at testing my latest this weekend, would anoyne be willing to help out with the windows version since I lack tools/machine etc?
best regards
Keith
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
John M McIntosh wrote:
(a) The OLPC actually uses a compressed file system already, because of that Published Sophie Books for the OLPC use a zipped file with no compression. The zipped file will live in the journal at some point.
(b) Does setting oops values back to start of zero actually make it more compressable, versus say a start value of 0x00004000?
We are not actually setting the oops back to zero, but back relative-to-zero, They are just numbers, just different numbers. I would not imagine that this has any effect upon the compressibility of any single image. However diff algorithms working with two or more images are more likely to see that much of a saved image is the same as the last time it was saved. From the little I know it seems that the OLPC file system (as well as mercurial binary storage) is designed with this in mind.
Indeed you make a good point, if we can pick a good number that runtimes can use without traversing the image then great. What number should it be?
Since OLPC applications have a virtual linux box to themselves can we just pick a number that suits us/them?
Otherwise the computational energy used to traverse the image is technically wasted.
(c) ittle/big endian? OLPC is based on the geod aka intel instruction set.
So it looks like little-endian is the favoured choice for the normalized image.
best regards
Keith
On Oct 5, 2007, at 5:05 , Keith Hodges wrote:
John M McIntosh wrote:
(a) The OLPC actually uses a compressed file system already, because of that Published Sophie Books for the OLPC use a zipped file with no compression. The zipped file will live in the journal at some point.
(b) Does setting oops values back to start of zero actually make it more compressable, versus say a start value of 0x00004000?
We are not actually setting the oops back to zero, but back relative-to-zero, They are just numbers, just different numbers. I would not imagine that this has any effect upon the compressibility of any single image. However diff algorithms working with two or more images are more likely to see that much of a saved image is the same as the last time it was saved. From the little I know it seems that the OLPC file system (as well as mercurial binary storage) is designed with this in mind.
Indeed you make a good point, if we can pick a good number that runtimes can use without traversing the image then great. What number should it be?
I don't think any OS (except for DOS maybe) would guarantee you a specific start address.
Since OLPC applications have a virtual linux box to themselves can we just pick a number that suits us/them?
I wouldn't think so ... but it is an interesting idea. I will ask the OLPC guys.
- Bert -
Bert Freudenberg schrieb:
On Oct 5, 2007, at 5:05 , Keith Hodges wrote:
Indeed you make a good point, if we can pick a good number that runtimes can use without traversing the image then great. What number should it be?
I don't think any OS (except for DOS maybe) would guarantee you a specific start address.
It does not need to guarantee a start address but just pick the same one most of the time (for example, as long as the VM binary stays the same). I think Squeak already does this. See below for another option.
Since OLPC applications have a virtual linux box to themselves can we just pick a number that suits us/them?
I wouldn't think so ... but it is an interesting idea. I will ask the OLPC guys.
It's not difficult for a VM to use mmap() on /dev/null (or was it /dev/zero?) to allocate memory for the image to any given address under linux (as long as the CPU/OS version supports memory at that address). If a VM did this, it could have a fixed image start address even across VM versions. Even cooler would be to mmap() the image file directly with copy-on-write behavior for almost instantaneous startup.
Cheers, Hans-Martin
On Fri, Oct 05, 2007 at 02:43:21PM +0200, Hans-Martin Mosner wrote:
Bert Freudenberg schrieb:
On Oct 5, 2007, at 5:05 , Keith Hodges wrote:
Indeed you make a good point, if we can pick a good number that runtimes can use without traversing the image then great. What number should it be?
I don't think any OS (except for DOS maybe) would guarantee you a specific start address.
It does not need to guarantee a start address but just pick the same one most of the time (for example, as long as the VM binary stays the same). I think Squeak already does this. See below for another option.
Since OLPC applications have a virtual linux box to themselves can we just pick a number that suits us/them?
I wouldn't think so ... but it is an interesting idea. I will ask the OLPC guys.
It's not difficult for a VM to use mmap() on /dev/null (or was it /dev/zero?) to allocate memory for the image to any given address under linux (as long as the CPU/OS version supports memory at that address). If a VM did this, it could have a fixed image start address even across VM versions. Even cooler would be to mmap() the image file directly with copy-on-write behavior for almost instantaneous startup.
Actually the Unix VM already does use mmap() on /dev/zero to obtain the base address, so it is easy to request a specific base address. Ian pointed this out to me when I was doing the "32bit clean" patches, and that is how I was able to test those changes.
The attached diff shows where to make the change in sqUnixMemory.c.
Note that mmap() will assign its own address if it does not agree with the one that you requested. You will need to pick an address that works on various versions of the kernel, and on both 32 bit and 64 bit systems (or perhaps provide a list of addresses, and let the VM find one that works on that particular platform). The value 0x4000000 seems reasonable on a 64 bit AMD Linux 2.6 system, but I did not check to see if it works on a 32 bit system, so you may need to experiment a bit.
The OLPC platform has well defined hardware and OS, so it would be good to pick a number that works on OLPC and make that the default.
Dave
On Oct 6, 2007, at 15:08 , David T. Lewis wrote:
On Fri, Oct 05, 2007 at 02:43:21PM +0200, Hans-Martin Mosner wrote:
Bert Freudenberg schrieb:
On Oct 5, 2007, at 5:05 , Keith Hodges wrote:
Indeed you make a good point, if we can pick a good number that runtimes can use without traversing the image then great. What number should it be?
I don't think any OS (except for DOS maybe) would guarantee you a specific start address.
It does not need to guarantee a start address but just pick the same one most of the time (for example, as long as the VM binary stays the same). I think Squeak already does this. See below for another option.
Since OLPC applications have a virtual linux box to themselves can we just pick a number that suits us/them?
I wouldn't think so ... but it is an interesting idea. I will ask the OLPC guys.
It's not difficult for a VM to use mmap() on /dev/null (or was it /dev/zero?) to allocate memory for the image to any given address under linux (as long as the CPU/OS version supports memory at that address). If a VM did this, it could have a fixed image start address even across VM versions. Even cooler would be to mmap() the image file directly with copy-on-write behavior for almost instantaneous startup.
Actually the Unix VM already does use mmap() on /dev/zero to obtain the base address, so it is easy to request a specific base address. Ian pointed this out to me when I was doing the "32bit clean" patches, and that is how I was able to test those changes.
The attached diff shows where to make the change in sqUnixMemory.c.
Note that mmap() will assign its own address if it does not agree with the one that you requested. You will need to pick an address that works on various versions of the kernel, and on both 32 bit and 64 bit systems (or perhaps provide a list of addresses, and let the VM find one that works on that particular platform). The value 0x4000000 seems reasonable on a 64 bit AMD Linux 2.6 system, but I did not check to see if it works on a 32 bit system, so you may need to experiment a bit.
The OLPC platform has well defined hardware and OS, so it would be good to pick a number that works on OLPC and make that the default.
Interesting - I also found that on Linux I can modify the executable to disable randomization using the -R flag of setarch:
http://www.linuxcommand.org/man_pages/setarch8.html
Have not tested this yet. And does the image loading logic actually check if a remap is necessary?
- Bert -
On Tue, Oct 09, 2007 at 10:26:42PM +0200, Bert Freudenberg wrote:
On Oct 6, 2007, at 15:08 , David T. Lewis wrote:
Actually the Unix VM already does use mmap() on /dev/zero to obtain the base address, so it is easy to request a specific base address. Ian pointed this out to me when I was doing the "32bit clean" patches, and that is how I was able to test those changes.
The attached diff shows where to make the change in sqUnixMemory.c.
Note that mmap() will assign its own address if it does not agree with the one that you requested. You will need to pick an address that works on various versions of the kernel, and on both 32 bit and 64 bit systems (or perhaps provide a list of addresses, and let the VM find one that works on that particular platform). The value 0x4000000 seems reasonable on a 64 bit AMD Linux 2.6 system, but I did not check to see if it works on a 32 bit system, so you may need to experiment a bit.
The OLPC platform has well defined hardware and OS, so it would be good to pick a number that works on OLPC and make that the default.
Interesting - I also found that on Linux I can modify the executable to disable randomization using the -R flag of setarch:
http://www.linuxcommand.org/man_pages/setarch8.html
Have not tested this yet. And does the image loading logic actually check if a remap is necessary?
I don't think there will be any need to use setarch. The VM justs asks for a big bunch of memory, and uses mmap() to do this. If you do not request a specific address to use in your process virtual memory, it will assign one for you. And if you do ask for an address, it will try to honor your request if it can. That's probably sufficient for our purposes as long as we can pick a reasonable address to request. If it turns out that there are lots of differences across platforms, it could be handled by configure (but hopefully that's not necessary).
Dave
Well if you're building a carbon os-x VM the build docs are in the readme in the mac os CVS tree
On Sep 29, 2007, at 12:56 PM, Keith Hodges wrote:
I am also unable to find any documentation on generating the .app version of the unix vm for Mac OS X.
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
vm-dev@lists.squeakfoundation.org