Dear all,
I'm just getting grips on the squeak image format and I am trying to make sense of the "512 bytes padding" possibility. In a previous discussion, the method Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset was mentioned that can read the image format using the given offset. However, neither in the Smalltalk code nor in the generated C code I was able to find any uses of this.
SqueakVMMaker/src ✔ grep -iR readImageFromFileHeapSizeStartingAt * vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset); vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
Is this by design? Did I miss something?
So Long, -Tobias
On 13/01/2011 16:26, Tobias Pape wrote:
Dear all,
I'm just getting grips on the squeak image format and I am trying to make sense of the "512 bytes padding" possibility. In a previous discussion, the method Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset was mentioned that can read the image format using the given offset. However, neither in the Smalltalk code nor in the generated C code I was able to find any uses of this.
SqueakVMMaker/src ✔ grep -iR readImageFromFileHeapSizeStartingAt * vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset); vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
Is this by design? Did I miss something?
So Long, -Tobias
Isn't this so that on unix systems you can make the image executable and put something like "#!/path/to/squeak/vm" at the start of the image so you can run it directly?
On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:
Dear all,
I'm just getting grips on the squeak image format and I am trying to make sense of the "512 bytes padding" possibility. In a previous discussion, the method Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset was mentioned that can read the image format using the given offset. However, neither in the Smalltalk code nor in the generated C code I was able to find any uses of this.
SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt * vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset); vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
Is this by design? Did I miss something?
So Long, -Tobias
The purpose of this feature is to support "shebang" processing: http://en.wikipedia.org/wiki/Shebang_(Unix)
The idea is to offset the start of the image by 512 bytes into the image file, and put the hash-bang line at the beginning of the file. The hash-bang line invokes the VM, and the VM reads the file containing the image at offset 512. The VM first tries to read the image starting at offset 0, which fails because it sees the hash-bang instead of the image format word. It then skips ahead 512 bytes, tries again, and presto the image starts up.
The end result is that an image file can be turned into an executable file that runs itself.
Dave
Am 2011-01-13 um 17:52 schrieb David T. Lewis:
On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:
[…] In a previous discussion, the method Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset was mentioned that can read the image format using the given offset. However, neither in the Smalltalk code nor in the generated C code I was able to find any uses of this.
SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt * vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset); vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
Is this by design? Did I miss something? […]
The purpose of this feature is to support "shebang" processing: http://en.wikipedia.org/wiki/Shebang_(Unix)
The idea is to offset the start of the image by 512 bytes into the image file, and put the hash-bang line at the beginning of the file. The hash-bang line invokes the VM, and the VM reads the file containing the image at offset 512. The VM first tries to read the image starting at offset 0, which fails because it sees the hash-bang instead of the image format word. It then skips ahead 512 bytes, tries again, and presto the image starts up.
The end result is that an image file can be turned into an executable file that runs itself.
The _purpose_ of the 512byte skip was clear to me. I was just puzzled that there is no reference to the indicated method whatsoever. Can you give me a hint, where that is used?
So Long, -Tobias
On Thu, Jan 13, 2011 at 06:20:28PM +0100, Tobias Pape wrote:
Am 2011-01-13 um 17:52 schrieb David T. Lewis:
On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:
[?] In a previous discussion, the method Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset was mentioned that can read the image format using the given offset. However, neither in the Smalltalk code nor in the generated C code I was able to find any uses of this.
SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt * vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset); vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
Is this by design? Did I miss something? [?]
The purpose of this feature is to support "shebang" processing: http://en.wikipedia.org/wiki/Shebang_(Unix)
The idea is to offset the start of the image by 512 bytes into the image file, and put the hash-bang line at the beginning of the file. The hash-bang line invokes the VM, and the VM reads the file containing the image at offset 512. The VM first tries to read the image starting at offset 0, which fails because it sees the hash-bang instead of the image format word. It then skips ahead 512 bytes, tries again, and presto the image starts up.
The end result is that an image file can be turned into an executable file that runs itself.
The _purpose_ of the 512byte skip was clear to me. I was just puzzled that there is no reference to the indicated method whatsoever. Can you give me a hint, where that is used?
I'm away from Squeak at the moment, but look in class Interpreter in package VMMaker. I think the method is #readImageFromFile:HeapSize:StartingAt: or similar. The colons get removed from the method/function name as part of the translation to C.
Dave
Am 2011-01-13 um 19:46 schrieb David T. Lewis:
On Thu, Jan 13, 2011 at 06:20:28PM +0100, Tobias Pape wrote:
Am 2011-01-13 um 17:52 schrieb David T. Lewis:
On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:
[?] In a previous discussion, the method Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize StartingAt: imageOffset was mentioned that can read the image format using the given offset. However, neither in the Smalltalk code nor in the generated C code I was able to find any uses of this.
SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt * vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset); vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
[…]
The _purpose_ of the 512byte skip was clear to me. I was just puzzled that there is no reference to the indicated method whatsoever. Can you give me a hint, where that is used?
I'm away from Squeak at the moment, but look in class Interpreter in package VMMaker. I think the method is #readImageFromFile:HeapSize:StartingAt: or similar. The colons get removed from the method/function name as part of the translation to C.
Yes, that is true. I've already found that, if you refer to my original post. My point is, that NEITHER the smalltalk method Interpreter>>#readImageFromFile:HeapSize:StartingAt: NOR the c function is called at all: No senders for the Smalltalk method, no reference when searching all methods source code for readImageFrom, aside the actual implementation and a mention in some ‘important methods’ collection. For the C-function, refer to my grep inside the generated source tree. No call at all.
So, why are they existent, in the first place?
So Long -Tobias
Hi,
Am 13.01.2011 um 20:57 schrieb Tobias Pape Das.Linux@gmx.de:
For the C-function, refer to my grep inside the generated source tree. No call at all.
just guessing: is there perhaps a call outside generated code, i.e., in common code? Anywhere in the whole bunch of sources?
Best,
Michael
On Thu, Jan 13, 2011 at 11:57 AM, Tobias Pape Das.Linux@gmx.de wrote:
Am 2011-01-13 um 19:46 schrieb David T. Lewis:
On Thu, Jan 13, 2011 at 06:20:28PM +0100, Tobias Pape wrote:
Am 2011-01-13 um 17:52 schrieb David T. Lewis:
On Thu, Jan 13, 2011 at 05:26:34PM +0100, Tobias Pape wrote:
[?] In a previous discussion, the method Interpreter>>readImageFromFile: f HeapSize: desiredHeapSize
StartingAt: imageOffset
was mentioned that can read the image format using the given offset. However, neither in the Smalltalk code nor in the generated C code I
was able
to find any uses of this.
SqueakVMMaker/src ??? grep -iR readImageFromFileHeapSizeStartingAt * vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f,
usqInt desiredHeapSize, squeakFileOffsetType imageOffset);
vm/interp.c:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f,
usqInt desiredHeapSize, squeakFileOffsetType imageOffset) {
[…]
The _purpose_ of the 512byte skip was clear to me. I was just puzzled that there is no reference to the indicated method
whatsoever.
Can you give me a hint, where that is used?
I'm away from Squeak at the moment, but look in class Interpreter in package VMMaker. I think the method is
#readImageFromFile:HeapSize:StartingAt:
or similar. The colons get removed from the method/function name as part of the translation to C.
Yes, that is true. I've already found that, if you refer to my original post. My point is, that NEITHER the smalltalk method Interpreter>>#readImageFromFile:HeapSize:StartingAt: NOR the c function
is called at all: No senders for the Smalltalk method, no reference when searching all methods source code for readImageFrom, aside the actual implementation and a mention in some ‘important methods’ collection. For the C-function, refer to my grep inside the generated source tree. No call at all.
So, why are they existent, in the first place?
You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. See platforms/{Mac OS,iOS,win32,unix}/vm
McStalker.oscogvm$ grep readImageFromFile platforms/*/vm/* platforms/Cross/vm/sq.h:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset); platforms/Cross/vm/sq.h:#define readImageFromFileHeapSize(f, s) readImageFromFileHeapSizeStartingAt(f,s,0) platforms/Mac OS/vm/sqMacMain.c: readImageFromFileHeapSizeStartingAt(f, sqGetAvailableMemory(), 0); platforms/Mac OS/vm/sqMacNSPluginUILogic.c: readImageFromFileHeapSizeStartingAt(f, squeakHeapMBytes, 0); platforms/unix/vm/sqUnixMain.c: readImageFromFileHeapSizeStartingAt(f, extraMemory, 0); platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, 0); platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, sqImageFilePosition(imageFile));
So Long -Tobias
Hi,
Am 13.01.2011 um 22:45 schrieb Eliot Miranda eliot.miranda@gmail.com:
You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. ...
:-)
OK. Say, *is* there comprehensive VM documentation other than in the form of code and some human minds?
Michael
On Thu, Jan 13, 2011 at 1:53 PM, Michael Haupt mhaupt@gmail.com wrote:
Hi,
Am 13.01.2011 um 22:45 schrieb Eliot Miranda eliot.miranda@gmail.com:
You're not searching all the C code ;)
readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. ...
:-)
OK. Say, *is* there comprehensive VM documentation other than in the form of code and some human minds?
No. The doc that we have form Tim, which is good, is out of date and doesn't describe Cog at all. I have yet to document the Cog design on my web site. There's a lot lacking in doc, but hopefully the minds are all present and correct :)
Michael
On Thu, Jan 13, 2011 at 02:04:11PM -0800, Eliot Miranda wrote:
On Thu, Jan 13, 2011 at 1:53 PM, Michael Haupt mhaupt@gmail.com wrote:
Hi,
Am 13.01.2011 um 22:45 schrieb Eliot Miranda eliot.miranda@gmail.com:
You're not searching all the C code ;)
readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. ...
:-)
OK. Say, *is* there comprehensive VM documentation other than in the form of code and some human minds?
No. The doc that we have form Tim, which is good, is out of date and doesn't describe Cog at all. I have yet to document the Cog design on my web site. There's a lot lacking in doc, but hopefully the minds are all present and correct :)
See also John's platform API documentation here: http://isqueak.org/PlatformVMAPI
Dave
We are planning to write some new chapters on vms so if people want to share effort they are welcomed. Our work will be hosted on pbe on github.
Stef
On Jan 13, 2011, at 10:53 PM, Michael Haupt wrote:
Hi,
Am 13.01.2011 um 22:45 schrieb Eliot Miranda eliot.miranda@gmail.com:
You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. ...
:-)
OK. Say, *is* there comprehensive VM documentation other than in the form of code and some human minds?
Michael
Am 2011-01-13 um 22:45 schrieb Eliot Miranda: […]
You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. See platforms/{Mac OS,iOS,win32,unix}/vm
Ok. I got that. Apologies, it was my fault, obviously.
McStalker.oscogvm$ grep readImageFromFile platforms/*/vm/* platforms/Cross/vm/sq.h:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset); platforms/Cross/vm/sq.h:#define readImageFromFileHeapSize(f, s) readImageFromFileHeapSizeStartingAt(f,s,0)
ok. the definitions.
platforms/Mac OS/vm/sqMacMain.c: readImageFromFileHeapSizeStartingAt(f, sqGetAvailableMemory(), 0); platforms/Mac OS/vm/sqMacNSPluginUILogic.c: readImageFromFileHeapSizeStartingAt(f, squeakHeapMBytes, 0);
So, the Carbon vm is _not_ dealing with the 512 offset?
platforms/unix/vm/sqUnixMain.c: readImageFromFileHeapSizeStartingAt(f, extraMemory, 0);
and, most curiously, the unix-VM isn't either?
platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, 0); platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, sqImageFilePosition(imageFile));
Ah, the Windows vm does.
I did one grep of my own and found:
vm has 512b-offset-support Carbon no Risc no Cocoa no unix no Win32 yes
So, while only the win32-vm is _able_ to read the offset images, wouldn't it make sense to drop this feature altogether?
So Long, -Tobias
On Fri, Jan 14, 2011 at 12:10:23PM +0100, Tobias Pape wrote:
Am 2011-01-13 um 22:45 schrieb Eliot Miranda: [?]
You're not searching all the C code ;) readImageFromFile:HeapSize:StartingAt: is called from the C main routine, and the C main routine is not generated. See platforms/{Mac OS,iOS,win32,unix}/vm
Ok. I got that. Apologies, it was my fault, obviously.
McStalker.oscogvm$ grep readImageFromFile platforms/*/vm/* platforms/Cross/vm/sq.h:sqInt readImageFromFileHeapSizeStartingAt(sqImageFile f, usqInt desiredHeapSize, squeakFileOffsetType imageOffset); platforms/Cross/vm/sq.h:#define readImageFromFileHeapSize(f, s) readImageFromFileHeapSizeStartingAt(f,s,0)
ok. the definitions.
platforms/Mac OS/vm/sqMacMain.c: readImageFromFileHeapSizeStartingAt(f, sqGetAvailableMemory(), 0); platforms/Mac OS/vm/sqMacNSPluginUILogic.c: readImageFromFileHeapSizeStartingAt(f, squeakHeapMBytes, 0);
So, the Carbon vm is _not_ dealing with the 512 offset?
platforms/unix/vm/sqUnixMain.c: readImageFromFileHeapSizeStartingAt(f, extraMemory, 0);
and, most curiously, the unix-VM isn't either?
platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, 0); platforms/win32/vm/sqWin32Intel.c: readImageFromFileHeapSizeStartingAt(imageFile, virtualMemory, sqImageFilePosition(imageFile));
Ah, the Windows vm does.
I did one grep of my own and found:
vm has 512b-offset-support Carbon no Risc no Cocoa no unix no Win32 yes
So, while only the win32-vm is _able_ to read the offset images, wouldn't it make sense to drop this feature altogether?
You're misreading the code. This feature is not platform specific, and is implemented in the Smalltalk code. This is explained in the method comment of #readImageFromFile:HeapSize:StartingAt: and you'll find the implementation in #checkImageVersionFrom:startingAt: which is sent by #readImageFromFile:HeapSize:StartingAt:
So main() calls #readImageFromFile:HeapSize:StartingAt: to load the image file, and the 512 byte skip is implemented there.
You can also see how this works by loading an image file in the InterpreterSimulator. See class comment for InterpreterSimulator, and step through it in a debugger to see your image file being loaded.
Dave
Am 2011-01-14 um 15:08 schrieb David T. Lewis:
On Fri, Jan 14, 2011 at 12:10:23PM +0100, Tobias Pape wrote:
[…]
Ah, the Windows vm does.
I did one grep of my own and found:
vm has 512b-offset-support Carbon no Risc no Cocoa no unix no Win32 yes
So, while only the win32-vm is _able_ to read the offset images, wouldn't it make sense to drop this feature altogether?
You're misreading the code. This feature is not platform specific, and is implemented in the Smalltalk code. This is explained in the method comment of #readImageFromFile:HeapSize:StartingAt: and you'll find the implementation in #checkImageVersionFrom:startingAt: which is sent by #readImageFromFile:HeapSize:StartingAt:
So main() calls #readImageFromFile:HeapSize:StartingAt: to load the image file, and the 512 byte skip is implemented there.
Ah, I see. Why is the windows vm doing it manually, then?
You can also see how this works by loading an image file in the InterpreterSimulator. See class comment for InterpreterSimulator, and step through it in a debugger to see your image file being loaded.
Nice hint. Thank you.
I did that on a hand-crafted 512-byte offset file. However, the InterpreterSimulator does not support 512byte offsetting:
InterpreterSimulator>>#openOn:extraMemory: " .. snip .."
["begin ensure block..." f := FileStream readOnlyFileNamed: fileName. imageName := f fullName. f binary. " offset handling should be here --> " version := self nextLongFrom: f. "current version: 16r1966 (=6502)" versionToRun := version bitAnd: -2. "permit loading images with platform float ordering" hasPlatformFloatOrdering := version ~= (version bitAnd: -2). "is low order bit set?" (self readableFormat: versionToRun) "permit loading images with platform float ordering" ifTrue: [swapBytes := false] ifFalse: [(versionToRun := self byteSwapped: version) = self imageFormatVersion ifTrue: [swapBytes := true] ifFalse: [ " or at least here -->" self error: 'incompatible image format']].
I tested the same Image on the Carbon 4.2.5beta1U VM, but it does not load either: /Applications/Squeak\ 4.2.5beta1U.app/Contents/MacOS/Squeak\ VM\ Opt v39_le_32bit_off_clos_cog.SqueakLight.image This interpreter (vers. 0) cannot read image file (vers. 1966022947). Press CR to quit...
Note that the 1966022947 = 0x752F2123 => 0x23 0x21 0x2F 0x75 corresponds to the beginning of the file: '#!/u'
Is anyone interested in the offset-image to test it her/himself?
So Long, -Tobias
On 1/20/2011 6:37 PM, Tobias Pape wrote:
So main() calls #readImageFromFile:HeapSize:StartingAt: to load the image file, and the 512 byte skip is implemented there.
Ah, I see. Why is the windows vm doing it manually, then?
Probably because nobody told me that that wasn't necessary :-) I'll happily remove it.
Cheers, - Andreas
vm-dev@lists.squeakfoundation.org
-
Andreas Raab -
David T. Lewis -
Douglas Brebner -
Eliot Miranda -
Michael Haupt -
stephane ducasse -
Tobias Pape