On Thu, Feb 12, 2009 at 9:15 AM, Eliot Miranda eliot.miranda@gmail.comwrote:

On Thu, Feb 12, 2009 at 1:32 AM, Andreas Raab andreas.raab@gmx.de wrote:

...

Folks -

Has anyone tried porting the FFI to 64 bit? I'm loosely considering using a 64bit VM for our servers but I do need the FFI (mostly for ODBC integration) and I've never looked at any of the ABIs. Alternatively, is it reasonable to try to use 64bit libs from within a 32bit Squeak VM?

Depends on what one wants to pass through the FFI. On x86-64/EMT64 structure passing conventions are efficient and complex, distributing structure fields across integer and floating-point registers. It is possible to interpret such calls given a type signature for the structure to be passed (I did it for VisualWorks) but it is not easy and it is slow. I would counsel writing a proper ABI compiler in Smalltalk that generates e.g. RTL that is either interpreted or compiled by the VM to make the actual call.

In theory the use of 64-bit libs, provided one has the relevant FFI support should work fine. Smalltalk has the ability to deal with 64-bit and 128-bit values easily. But one does have to write all the support.

I should expand on this, and could be completely wrong. One would have to use a 64-bit address space, which implies compiling the 32-bit VM using a 64-bit compiler. When DEC and ParcPlace co-implemented 32-bit VisualWorks (my first job on joining ParcPlace was to productise this port) we used the DEC C compiler's ability to define "short" (32-bit) pointers and put the executable in the lower 32-bits of the address space (google for e.g. -xtaso or taso). Hence the VM was a 64-bit executable which had a 32-bit Smalltalk living in the bottom 32-bits. If x86-64 compilers have the same short pointer support then the compilation of a 32-bit Squeak VM should be relatively easy. Otherwise one will have to define the longAt: longAt:put: et al machinery to extend 32-bit addresses into full 64-bit pointers and genuflect to the linker and/or memory allocator to get it to place the heap low in the address space.

To get acquainted with the de facto standard x86-64 ABI read the System V Application Binary Interface AMD64 Architecture Processor Supplement http://www.x86-64.org/documentation/abi-0.98.pdf pages 13 through 22 (section Function Calling Sequence) paying particular attention to the 3 1/2 page algorithm for marshalling :)

and start going to bed earlier ;)

...

Thanks for any info!

Cheers,

• Andreas

On Thu, Feb 12, 2009 at 9:43 AM, Bert Freudenberg bert@freudenbergs.dewrote:

On 12.02.2009, at 18:22, Eliot Miranda wrote:

...

I should expand on this, and could be completely wrong.

Since you're late to the game I'll point out

http://squeakvm.org/squeak64/faq.html

I see that Bert, but Andreas is asking something different. He is asking if the existing 32-bit system can be run on a 64-bit platform, not that one can run a 64-bit VM on a 64-bit platform. In Qwaq we have 32-bit server images and Andreas is interested in using these images largely unchanged to run as 64-bit processes.

Cheers Eliot

http://squeakvm.org/squeak64/faq.html

• Bert -

On Thu, Feb 12, 2009 at 9:53 AM, Bert Freudenberg bert@freudenbergs.dewrote:

On 12.02.2009, at 18:48, Eliot Miranda wrote:

...

On Thu, Feb 12, 2009 at 9:43 AM, Bert Freudenberg bert@freudenbergs.de wrote:

On 12.02.2009, at 18:22, Eliot Miranda wrote:

I should expand on this, and could be completely wrong.

Since you're late to the game I'll point out

http://squeakvm.org/squeak64/faq.html

I see that Bert, but Andreas is asking something different. He is asking if the existing 32-bit system can be run on a 64-bit platform, not that one can run a 64-bit VM on a 64-bit platform. In Qwaq we have 32-bit server images and Andreas is interested in using these images largely unchanged to run as 64-bit processes.

... which is what the 64 bit VM does, as Andreas is well-aware. One of the remaining problems is 64 bit FFI, and a few more plugins. Fortunately David has been working on those for quite some time.

IIUC the 64-bit VM does _not_ run 32-bit images. So I don't understand how the 64-bit VM is releant. Can yu lead me by the nose a little further?

TIA

• Bert -

On Thu, Feb 12, 2009 at 10:20 AM, Bert Freudenberg bert@freudenbergs.dewrote:

On 12.02.2009, at 19:00, Eliot Miranda wrote:

...

I see that Bert, but Andreas is asking something different. He is asking

...

...

if the existing 32-bit system can be run on a 64-bit platform, not that one can run a 64-bit VM on a 64-bit platform. In Qwaq we have 32-bit server images and Andreas is interested in using these images largely unchanged to run as 64-bit processes.

... which is what the 64 bit VM does, as Andreas is well-aware. One of

the remaining problems is 64 bit FFI, and a few more plugins. Fortunately David has been working on those for quite some time.

IIUC the 64-bit VM does _not_ run 32-bit images.

It does.

So I don't understand how the 64-bit VM is releant. Can yu lead me by

...

the nose a little further?

Sure. *Please* read the FAQ:

Um, I have (studied my Agrippa), and I quote:

Does my 64-bit VM run both 32-bit and 64-bit images?No. Any VM will run either 32-bit or 64-bit images, but not both. You can select one or the other when you generate sources with VMMaker, and you can install both flavors of VM on your system (one each for 32-bit images and 64-bit images).

If you try to run a 64-bit image with a VM built for 32-bit images, you will get an error message such as this:

This interpreter (vers. 6502) cannot read image file (vers. 68000).

If you try to run a 32-bit image using a VM built for 64-bit images, you will get an error message such as this:

This interpreter (vers. 68000) cannot read image file (vers. 6502).

So no, one cannot run a 32-bit image in a 64-bit VM (as I had thought) and the FAQ is quite specific about it. One has to convert a 32-bit image to a 64-bit image to run it on the 64-bit machine (as detailed ijn the FAQ) and that is not what Andreas is interested in doing.

Apologies for being so persistent :)

http://squeakvm.org/squeak64/faq.html

...

• Bert -

Eliot Miranda wrote:

Um, I have (studied my Agrippa), and I quote:

Does my 64-bit VM run both 32-bit and 64-bit images? No. Any VM will run either 32-bit or 64-bit images, but not both.

This is a bit misleading. What it means is that once you've decided whether a 64 bit VM will be used with 32 or 64 bit images, you cannot change it. However, 64 bit VMs can absolutely be used to run 32 bit images (and why wouldn't they).

Cheers, - Andreas

You can select one or the other when you generate sources with
VMMaker, and you can install both flavors of VM on your system (one
each for 32-bit images and 64-bit images).

If you try to run a 64-bit image with a VM built for 32-bit images,
you will get an error message such as this:

    This interpreter (vers. 6502) cannot read image file (vers. 68000).

If you try to run a 32-bit image using a VM built for 64-bit images,
you will get an error message such as this:

    This interpreter (vers. 68000) cannot read image file (vers. 6502).

So no, one cannot run a 32-bit image in a 64-bit VM (as I had thought) and the FAQ is quite specific about it. One has to convert a 32-bit image to a 64-bit image to run it on the 64-bit machine (as detailed ijn the FAQ) and that is not what Andreas is interested in doing.

Apologies for being so persistent :)

    http://squeakvm.org/squeak64/faq.html



- Bert -

Eliot Miranda wrote:

I'm assuming that in-memory the 64-bit VM has 64-bit oops, right? So the VM has to swizzle 32-bit pointers to 64-bit. I haven't come across this swizzling code yet. I thought that the swizzling was done by the system tracer. Where is the in-VM code to swizzle pointers to 64-bits?

oopForPointer/pointerForOop translate between OOPs and pointers. So you can translate from a 32bit Oop into a 64bit pointer by adding a base address and vice versa.

Cheers, - Andreas

Eliot Miranda wrote:

So I don't see how this allows one to have a 64-bit VM that can run true 64-bit images. What am I missing?

You are mostly confused about the various combinations ;-) Let's start from first principles: You can run 32bit images on top of 64 bit VMs. This is achieved by using oopForPointer/pointerForOop.

In order to run 64 bit images you need a 64 bit image to begin with. The 64 bit VM doesn't auto-magically convert your 32bit image into a 64 bit one. This is done by SystemTracer and friends.

Does that help?

Cheers, - Andreas

On Feb 12, 2009, at 11:30 AM, Eliot Miranda wrote:

There's a dearth of comment from Dan and Ian and David who ave actually worked with the 64-bit VM.

Well, Bert and Andreas and Dave are explaining it admirably well. Anything I might add would be redundant at best and confusing at worst.

Of course any explanation that makes perfect sense to me might not make any sense whatsoever to someone who is not already in possession of the knowledge that they're trying to communicate.

Cheers, Ian

On Thu, Feb 12, 2009 at 11:18:02AM -0800, Eliot Miranda wrote:

AFAICT you can produce a 64-bit VM that runs a 32-bit image with this technique, but you can't get a VM that runs both.

Actually, I think that it probably *is* possible to make a Squeak VM that will run both 32 and 64 bit images, although performance may be poor and the plugins would be problematic. The basic idea is to move BytesPerWord and BaseHeaderSize into run time variables rather than compile time macros (i.e. move them from class variables in ObjectMemory to instance variables in ObjectMemory), and set the values at image load time. Everything else should just work.

I've been meaning to do this for quite some time but have not gotten around to trying it.

Dave

On Thu, Feb 12, 2009 at 09:47:44PM -0800, Eliot Miranda wrote:

On Thu, Feb 12, 2009 at 7:38 PM, David T. Lewis lewis@mail.msen.com wrote:

...

On Thu, Feb 12, 2009 at 11:18:02AM -0800, Eliot Miranda wrote:

...

AFAICT you can produce a 64-bit VM that runs a 32-bit image with this technique, but you can't get a VM that runs both.

Actually, I think that it probably *is* possible to make a Squeak VM that will run both 32 and 64 bit images, although performance may be poor and the plugins would be problematic. The basic idea is to move BytesPerWord and BaseHeaderSize into run time variables rather than compile time macros (i.e. move them from class variables in ObjectMemory to instance variables in ObjectMemory), and set the values at image load time. Everything else should just work.

You'll have to change oopForPointer/pointerForOop to account for the different widths of an oop. e.g. if the heap is origined at 0 then in a 64-bit image the oop 4 maps to byte address 8, but in a 32-bit image it maps to byte address 4.

You'll have to change the readers and writers (e.g. oopAtPointer/oopAtPointerPut) to test BytesPerWord and fetch/store 64 or 32 bits as required.

Piece o' cake. I bet you a virtual beer I can get it working without touching any of these ;)

It won't be particularly sprightly :)

No wagers on that one. It would be interesting to see just how much slower it is though. Just to hazard a guess, I would expect about a 30% hit. If I get it working, I'll report back on results.

Dave

Eliot Miranda writes:

On Thu, Feb 12, 2009 at 7:38 PM, David T. Lewis lewis@mail.msen.com wrote:

...

On Thu, Feb 12, 2009 at 11:18:02AM -0800, Eliot Miranda wrote:

...

AFAICT you can produce a 64-bit VM that runs a 32-bit image with this technique, but you can't get a VM that runs both.

Actually, I think that it probably *is* possible to make a Squeak VM that will run both 32 and 64 bit images, although performance may be poor and the plugins would be problematic. The basic idea is to move BytesPerWord and BaseHeaderSize into run time variables rather than compile time macros (i.e. move them from class variables in ObjectMemory to instance variables in ObjectMemory), and set the values at image load time. Everything else should just work.

You'll have to change oopForPointer/pointerForOop to account for the different widths of an oop. e.g. if the heap is origined at 0 then in a 64-bit image the oop 4 maps to byte address 8, but in a 32-bit image it maps to byte address 4.

You'll have to change the readers and writers (e.g. oopAtPointer/oopAtPointerPut) to test BytesPerWord and fetch/store 64 or 32 bits as required.

At least.

It won't be particularly sprightly :)

Though, for those of us working on JIT compilers the checks will move back to compile time again.

Bryce

On Thu, Feb 12, 2009 at 11:25 AM, Bert Freudenberg bert@freudenbergs.dewrote:

On 12.02.2009, at 19:29, Eliot Miranda wrote:

...

Um, I have (studied my Agrippa), and I quote:

Does my 64-bit VM run both 32-bit and 64-bit images? No. Any VM will run either 32-bit or 64-bit images

Read it like this:

Does my 64-bit VM run *both* 32-bit and 64-bit images? No. Any VM will run *either* 32-bit *or* 64-bit images

Apologies for being so persistent :)

...

Well everyone has a thick day once in a while.

http://squeakvm.org/squeak64/faq.html

...

You still fail to see the crucial distinction the FAQ makes between 64-bit *images* and 64-bit *VMs*:

What is a 64-bit image? A 64-bit image is an image in which the object memory uses a 64-bit word size for object pointers

What is a 64-bit VM? A 64-bit VM is one which is compiled with the LP64 or ILP64 data model. This means, in C terms, that pointers and longs are 64-bits wide.

These two are independent. There are four kinds of VMs:

1. a 32-bit VM compiled to run 32-bit images
2. a 32-bit VM compiled to run 64-bit images
3. a 64-bit VM compiled to run 32-bit images
4. a 64-bit VM compiled to run 64-bit images

Wether the VM can run 32 or 64 bit *images* is governed by the checkbox in VMMaker. Wether the VM can run on a 32 or 64 bit host is governed by the C compiler.

Also, nobody uses 64 bit images yet.

Please suggest how the FAQ could make this more clear (assuming you actually studied the whole thing and not just skimmed it).

Speaking through a self-imposed gag let me suggest the FAQ distinguish clearly between a 64-bit VM (meaning a VM that runs 64-bit images that necessarly is compiled as a 64-bit executable) and a 32-bit VM compiled to run as a 64-bit executable. The latter doesn't count as a 64-bit VM for me. 64-bit clean perhaps.

Let me further suggest that 2) is pointless, in that it can't scale beyond the 32-bit address space but uses about 1.5 times the space of a 32-bit image as it tries to do so. i.e. it'll fail a lot earlier than a 32-bit image. Further, if a 64-bit image is larger than will fit in a 32-bit address space one *cant* run it on 2 because it won't fit. However, one can derive a 32-bit image from a 64-bit image that isn't too big using the SystemTracer (don't know if tis has been implemented; we only implemented 32-bit to 64-bit conversion in VsualWorks, uing something akin to the system tracer).

So I would suggest the FAQ explain tat there are really three different VMs, two 32-bit ones and one 64-bit one. One of the 32-bit VMs is compiled to run as a 64-bit executable on 64-bit systems but only runs a 32-bit image. That at least accords with the planet I grew up on :)

• Bert -

On Thu, Feb 12, 2009 at 11:55 AM, Andreas Raab andreas.raab@gmx.de wrote:

Eliot Miranda wrote:

...

Let me further suggest that 2) is pointless, in that it can't scale beyond the 32-bit address space but uses about 1.5 times the space of a 32-bit image as it tries to do so. i.e. it'll fail a lot earlier than a 32-bit image. Further, if a 64-bit image is larger than will fit in a 32-bit address space one *cant* run it on 2 because it won't fit.

It is not pointless at all. If we were to switch our servers to 64 bit images while allowing customers to deploy on 32bit, we'd be nuts trying to manage both 32 and 64 bit versions of the same image and trying to keep them in sync. Much easier to give them a 32bit VM that can run the very same 64 bit image that we use everywhere.

I respectfully disagree. The VisualWorks 32-bit to 64-bit converter produces something one can deploy immediately. So its not difficult to keep the two in sync. One merely passes the 32-bit image through the converter and fires up the 64-bit image. Saving the 64-bit image can save you a bit of start-up time, so is worth-while doing. Both these processes can be wrapped up in a shell script and run without user intervention. So to keep the 64-bit image in sync with a 32-bit one is easy. So 2) still strikes me as pointless.

In addition, it is this very combination that allows people without 64bit

boxes to actually help in working out the quirks with the 64bit images. I don't think we would have gotten where we are without it.

I don't want to be sarcastic, but that doesn't seem to be very far. From Bert I get the sense that no-one is actually using 64-bit Squeak images (he just said as much). The Squeak 64-bit image format is not particularly ambitious. It doesn't give one wider SmallInteger range, more identity hash bits, larger objects, more literals in a method or faster floats, or immediate characters. It is as naive a conversion to 64-bits as one can imagine. I don't see that the rather conservative approach taken with the 64-bit VM has produced something particularly compelling. Note that a memory-bound 64-bit Smalltalk application that can't take advantage of 64-bits to speed-up computation (as for example immediate doubles do) simply uses twice the memory bandwidth to accomplish exactly the same thing as a 32-bit application, and so if memory-limited (doesn't fit in cache) can run twice as slow. The only advantage such a 64-bit system has is in being ale to instantiate more objects.

VisualWorks' 64-bit implementation on the other hand provides immediate doubles that have performance only 1/2 that of SmallIntegers, provides 61-bit SmallIntegers and 20-bit id hashes vs 14 bits, much faster at: and at:put:, and is being used by customers using images > 4Gb in production. I think its got a lot further, and it didn't waste effort with 2) in doing so. So that's an existence proof that 2) doesn't help.

Again Im not trying to be offensive or toot my own horn. I'm just trying to be objective.

Regards, Eliot

Cheers,

• Andreas

On 12.02.2009, at 20:40, Eliot Miranda wrote:

...

1. a 32-bit VM compiled to run 32-bit images
2. a 32-bit VM compiled to run 64-bit images
3. a 64-bit VM compiled to run 32-bit images
4. a 64-bit VM compiled to run 64-bit images

Speaking through a self-imposed gag let me suggest the FAQ distinguish clearly between a 64-bit VM (meaning a VM that runs 64- bit images that necessarly is compiled as a 64-bit executable) and a 32-bit VM compiled to run as a 64-bit executable. The latter doesn't count as a 64-bit VM for me. 64-bit clean perhaps.

It may not count for you but that's how we have been calling it. Since nobody expects 64 bit images to work "64-bit" VM refers to the host architecture. I'm not quite sure how we could phase out that nomenclature.

Let me further suggest that 2) is pointless, in that it can't scale beyond the 32-bit address space but uses about 1.5 times the space of a 32-bit image as it tries to do so. i.e. it'll fail a lot earlier than a 32-bit image. Further, if a 64-bit image is larger than will fit in a 32-bit address space one *cant* run it on 2 because it won't fit.

The point of this was not to be used in production but to *create* VM for running a 64 bit image without having to use a 64 bit host.

So I would suggest the FAQ explain tat there are really three different VMs, two 32-bit ones and one 64-bit one. One of the 32- bit VMs is compiled to run as a 64-bit executable on 64-bit systems but only runs a 32-bit image. That at least accords with the planet I grew up on :)

Only if you want to use 64 bit images. Which nobody wants. Yet.

- Bert -

On Thu, Feb 12, 2009 at 10:55 AM, Andreas Raab andreas.raab@gmx.de wrote:

Eliot Miranda wrote:

...

I see that Bert, but Andreas is asking something different. He is asking if the existing 32-bit system can be run on a 64-bit platform, not that one can run a 64-bit VM on a 64-bit platform. In Qwaq we have 32-bit server images and Andreas is interested in using these images largely unchanged to run as 64-bit processes.

Well, actually, I am asking both ;-) The real problem is elsewhere. We have two issues that we need to address, one is the need for 32 bit libraries on a 64 bit system[*] and the other one is the need for utilization of more memory. The latter is a minor issue though, the real problem is the former. Because of this, I really don't care whether the image is 32 or 64 bit, but I *do* care whether the library the VM links against and that the plugins (FFI above all else) will work in the 64bit environment.

So in short, a 64bit VM is a must, a 64bit image somewhat optional.

[*] Really, the problem is conflicts between the two. Some libraries cannot have both 32 and 64 bit versions installed on the same box and unfortunately ODBC (which we rely heavily on) is one of those.

That's funny. This is on linux right? I always thought that the linux scheme of keeping the old names /lib, /usr/lib etc 32-bit and adding /lib64 /usr/lib64 was a neat way of allowing old 32-bit binaries to run unchanged. So where is the conflict with the ODBC libraries?

Cheers,

• Andreas

On Thu, Feb 12, 2009 at 10:55:51AM -0800, Andreas Raab wrote:

Well, actually, I am asking both ;-) The real problem is elsewhere. We have two issues that we need to address, one is the need for 32 bit libraries on a 64 bit system[*] and the other one is the need for utilization of more memory. The latter is a minor issue though, the real problem is the former. Because of this, I really don't care whether the image is 32 or 64 bit, but I *do* care whether the library the VM links against and that the plugins (FFI above all else) will work in the 64bit environment.

To the best of my knowledge, on 64 bit Linux there is no way for a 64 bit program to call a library compiled for 32 bit. See for example http://forums.amd.com/devblog/blogpost.cfm?threadid=93648&catid=317

It might be possible to extend the FFI implementation in some way that would allow a 64 bit VM to set up the call stack for a 32 bit library, but I for one certainly do not know how to do this.

So in short, a 64bit VM is a must, a 64bit image somewhat optional.

[*] Really, the problem is conflicts between the two. Some libraries cannot have both 32 and 64 bit versions installed on the same box and unfortunately ODBC (which we rely heavily on) is one of those.

If you just need a 64 bit VM calling various 64 bit libraries, then the problem is straightforward, and you (or we) just need to do Mantis 7237. If you also need to call some 32 bit libraries, then the problem is more challenging, and I do not know of a solution (*).

Dave

(*) If a vendor cannot or will not support their 32 bit and 64 bit ODBC libraries on a single host, I would look for some creative unix hacker to think of a workaround. There is sure to be some way to rename things or chroot or otherwise trick the system into doing what you want, even if the vendor is obstinate or clueless.