[Vm-dev] [squeak-dev] 64 bit FFI (was: porting Croquet to Squeak6.0 alpha...)

Marcel Taeumel marcel.taeumel at hpi.de
Tue Jun 9 16:40:14 UTC 2020


Hi Eliot.

> It might be better to use a C name such as intptr_t or uintptr_t.

Sounds good. :-)

Best,
Marcel
Am 09.06.2020 18:33:36 schrieb Eliot Miranda <eliot.miranda at gmail.com>:

On Jun 9, 2020, at 5:10 AM, Marcel Taeumel <marcel.taeumel at hpi.de> wrote:









Hi Nicolas, hi Eliot.

> If you want a type that can hold a pointer on both 32 and 64bits arch, then it's doable with a FFI type alias

You can just make an alias to a void*. It will have the right size. The only drawback would be that you might get an issue with argument coercion.

#(nil 'void*')


Your "type alias struct" will wrap around ExternalData<void*>. Why bother with ulong/ulonglong. Only if you want to work with integers in the image, you might want to use ulong or ulonglong depending on the #wordSize.

I think I will add 'word' as a type name in ExternalType's class-side "type constants" in Squeak FFI. Then you can use it in FFI call defs and struct-field defs:

ExternalType class >> #word
"Use this atomic-type alias if the external library works with opaque handles that are different on 32bit and 64bit. On a platform change, all FFI calls will be re-compiled and this method is called again by the parser."
^ FFIPlatformDescription current wordSize = 4
ifTrue: [self unsignedLong]
ifFalse: [self unsignedLongLong]

... maybe parts in the FFI plugin are faster with ulong/ulonglong compared to void* compared to type alias to void*... I don't know.

It might be better to use a C name such as intptr_t or uintptr_t.


Best,
Marcel



Am 30.05.2020 18:00:33 schrieb Marcel Taeumel <marcel.taeumel at hpi.de>:






So, if the atomic types and structure types are initialized, they make use of ExternalType class >> #pointerSpec to store the size of the pointer in the #headerWord, which is the first element of an external type's 'compiledSpec'. For atomic types, this happens when the FFI package is loaded.

For structure types, this happens when a an external structure is mentioned for the first time in an FFI call method or in during compilation of another external structure's fields. See ExternalType class >> #newTypeNamed:force:. Squeak's Smalltalk parser or the compile logic in ExternalStructure invokes that type creation respectively..

Currently, if a platform change at system startup is detected, all atomic types and structure types are updated --- which implies a call to #pointerSpec in ExternalType class >> #initializeStructureTypes.

Now, what could be the situation where a struct-pointer type had the wrong pointer size? During development, your platform does not magically change the #wordSize. When loading code, methods in the external struct might be wrong (bc. Monticello does not trigger that field re-definition) BUT any external struct-pointer type that is already in the image must have the right size. And for new external struct-pointer types, it should work as explained above.

So, I think we should make the following changes:
- Remove ExternalType >> #asPointerType: (! note the colon !)
- Remove ExternalType >> #pointerSize:
- Change ExternalType >> #pointerSize to just read the compiledSpec
- Update all sends to #asPointerType: to #asPointerType and assume that the current pointer-type instances have the right size, as described above.

Best,
Marcel




Am 30.05.2020 15:59:31 schrieb Marcel Taeumel <marcel.taeumel at hpi.de>:

> But if it is obsolete now, nuke it.

I cannot tell. That information is stored in the external type's compiledSpec to be used from within the FFI plugin I suppose.

Best,
Marcel

Am 29.05.2020 00:15:25 schrieb Eliot Miranda <eliot.miranda at gmail.com>:
On Thu, May 28, 2020 at 11:33 AM Nicolas Cellier <>
nicolas.cellier.aka.nice at gmail.com> wrote:



>

> Hi Marcel,

> no idea...

> Look at timestamp:

> eem 2/16/2016 12:42 · accessing · 2 implementors · only in change set

> FFI-Kernel-EstebanLorenzano.45 ·

> So presumably Eliot added that.

>



Perhaps. I don't remember. But if it is obsolete now, nuke it.





>

> Le jeu. 28 mai 2020 à 17:00, marcel.taeumel a

> écrit :

>

>>

>> Hi Nicolas,

>>

>> what is the role of ExternalStructure class >> #pointerSize then?

>>

>> Best,

>> Marcel

>>

>>

>> Nicolas Cellier wrote

>> > Hi all,

>> > beware of FFI 'long' and 'ulong'.

>> > They are currently meaning (32 bits) int and unsigned int!

>> > This dates back from the 80s when there were still odd OS with segmented

>> > memory and 16-bits int.

>> > long was thought as a safe int32_t at that time... and still means just

>> > that in current FFI.

>> >

>> > I'd like to replace them with 'int' and 'uint', but we need to ensure

>> > backward compatibility etc...

>> >

>> > FFI has ExternalType signedLongLong or just 'longlong' and ExternalType

>> > unsignedLongLong or 'ulonglong' which are 64 bits on all architectures.

>> > See implementors of #initializeAtomicTypes #initializeFFIConstants

>> >

>> > If you want a type that can hold a pointer on both 32 and 64bits arch,

>> > then

>> > it's doable with a FFI type alias

>> > For example I have things like this in HDF5 module

>> > Notice the single #(fieldName fieldType) which defines an alias, while

>> > ordinary FFI struct gets an array of #(fieldName fieldType)

>> > (and replace Size_t with Intptr_t if the purpose is hodling a pointer,

>> > otherwise the ayatollahs of C standard will lash you)

>> >

>> > ExternalStructure subclass: #'Size_t'

>> > instanceVariableNames: ''

>> > classVariableNames: ''

>> > poolDictionaries: ''

>> > category: 'HDF5-External-atomic'

>> >

>> > Size_t class >>fields

>> > "Size_t defineFields"

>> > ^Smalltalk wordSize = 4

>> > ifTrue: [#( value 'ulong')]

>> > ifFalse: [#( value 'ulonglong')]

>> >

>> > However the "fields" must be re-generated when platform changes...

>> > I tried to automate that sometimes ago, see:

>> >

>> > ExternalStructure class>>install

>> > "Resuming the image on another architecture may require a

>> > re-compilation of structure layout."

>> > | newPlatform |

>> > newPlatform := Smalltalk platformName.

>> > PreviousPlatform = newPlatform

>> > ifFalse:

>> > [self recompileStructures.

>> > PreviousPlatform := newPlatform]

>> >

>> > It works if you switch from Win64 to MacOS64 for example...

>> > However, this mechanism is not used yet at time of code loading, so

>> > importing code generated on a different platform won't automatically

>> > update

>> > the structures (or alias) - the FFI package has no mechanism for that

>> > (maybe we could add a hook in MC?).

>> >

>> > Le mer. 11 mars 2020 à 23:58, Bert Freudenberg <

>>

>> > bert@

>>

>> > > a

>> > écrit :

>> >

>> >> I'd suggest to get OpenGL working outside of Croquet first:

>> >>

>> >> http://www.squeaksource.com/CroquetGL.html

>> >>

>> >> Step 1: Verify this works in 32 bits. (assuming you are doing this on

>> >> Linux, you can run 32 bit Squeak side-by-side with the 64 bit one)

>> >> Step 2: Make it work in 64 bits.

>> >>

>> >> The second step requires that you understand how FFI works, and how it

>> >> handles e.g. pointers and integer sizes.

>> >> I am assuming we do have a working 64 bit FFI, at least for x86_64

>> >> machines.

>> >>

>> >> E.g. the OGLUnix>>glExtGetProcAddress: method returns a pointer. On a

>> 32

>> >> bit system, that fits into a 'ulong' which is 32 bits. On a 64 bit

>> >> system,

>> >> a pointer is 64 bits wide so it would not fit into a 32 bit word. Now I

>> >> don't know how many bits 'ulong' has in our 64 bit FFI, but that

>> >> declaration may have to change. Etc. pp.

>> >>

>> >> If you have questions about FFI then those are best directed at the

>> >> vm-dev

>> >> list since it is dealing with VM-level interfaces. CC'ing, please

>> follow

>> >> up

>> >> there.

>> >>

>> >> - Bert -

>> >>

>> >> On Wed, Mar 11, 2020 at 2:54 PM gettimothy via Squeak-dev <>
>> >>

>>

>> > squeak-dev at .squeakfoundation

>>

>> >> wrote:

>> >>

>> >>> Okey dokey,

>> >>>

>> >>> Poking along, there is a stray glyph in OGLUnix openGLLibraryName

>> after

>> >>>

>> >>> openGLLibraryName

>> >>> ^Smalltalk osVersion = 'linux'

>> >>> ifTrue: ['libGL.so.1']

>> >>> ifFalse: ['GL']

>> >>>

>> >>> I removed it in my install and got past that error.

>> >>>

>> >>> Working exclusively with Croquet(Master)...

>> >>>

>> >>>

>> >>>

>> >>> My next error is in OGLUnixX11LE(OpenGL)>>glMatrixMode:

>> >>>

>> >>> glMatrixMode: mode

>> >>> "This method was automatically generated."

>> >>> "void glMatrixMode(GLenum mode);"

>> >>>

>> >

>> >>> ^self externalCallFailed

>> >>>

>> >>> The

>> >

>> > fails

>> >>>

>> >>> How to think about this?

>> >>>

>> >>> Is Croquet behind OpenGL latest?

>> >>> Would teaching myself OpenGL programming be of use to the Croquet

>> >>> project?

>> >>>

>> >>> cheers,

>> >>>

>> >>> tty

>> >>>

>> >>>

>> >>>

>> >>

>>

>>

>>

>>

>>

>> --

>> Sent from: http://forum.world.st/Squeak-VM-f104410.html

>>

>



--

_,,,^..^,,,_

best, Eliot




On Thu, May 28, 2020 at 11:33 AM Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com [mailto:nicolas.cellier.aka.nice at gmail.com]> wrote:

 
Hi Marcel,
no idea...
Look at timestamp:
eem 2/16/2016 12:42 · accessing · 2 implementors · only in change set FFI-Kernel-EstebanLorenzano.45  ·
So presumably Eliot added that.


Perhaps.  I don't remember.  But if it is obsolete now, nuke it.
 

Le jeu. 28 mai 2020 à 17:00, marcel.taeumel <Marcel.Taeumel at hpi.de [mailto:Marcel.Taeumel at hpi.de]> a écrit :

 


Hi Nicolas,





what is the role of ExternalStructure class >> #pointerSize then?





Best,


Marcel








Nicolas Cellier wrote


> Hi all,


> beware of FFI 'long' and 'ulong'.


> They are currently meaning (32 bits) int and unsigned int!


> This dates back from the 80s when there were still odd OS with segmented


> memory and 16-bits int.


> long was thought as a safe int32_t at that time... and still means just


> that in current FFI.


>


> I'd like to replace them with 'int' and 'uint', but we need to ensure


> backward compatibility etc...


>


> FFI has ExternalType signedLongLong or just 'longlong' and ExternalType


> unsignedLongLong or 'ulonglong' which are 64 bits on all architectures.


> See implementors of #initializeAtomicTypes #initializeFFIConstants


>


> If you want a type that can hold a pointer on both 32 and 64bits arch,


> then


> it's doable with a FFI type alias


> For example I have things like this in HDF5 module


> Notice the single #(fieldName fieldType) which defines an alias, while


> ordinary FFI struct gets an array of #(fieldName fieldType)


> (and replace Size_t with Intptr_t if the purpose is hodling a pointer,


> otherwise the ayatollahs of C standard will lash you)


>


> ExternalStructure subclass: #'Size_t'


>     instanceVariableNames: ''


>     classVariableNames: ''


>     poolDictionaries: ''


>     category: 'HDF5-External-atomic'


>


> Size_t class >>fields


>     "Size_t defineFields"


>     ^Smalltalk wordSize = 4


>         ifTrue: [#( value 'ulong')]


>         ifFalse: [#( value 'ulonglong')]


>


> However the "fields" must be re-generated when platform changes...


> I tried to automate that sometimes ago, see:


>


> ExternalStructure class>>install


>      "Resuming the image on another architecture may require a


> re-compilation of structure layout."


>      | newPlatform |


>      newPlatform := Smalltalk platformName.


>      PreviousPlatform = newPlatform


>          ifFalse:


>              [self recompileStructures.


>             PreviousPlatform := newPlatform]


>


> It works if you switch from Win64 to MacOS64 for example...


> However, this mechanism is not used yet at time of code loading, so


> importing code generated on a different platform won't automatically


> update


> the structures (or alias) - the FFI package has no mechanism for that


> (maybe we could add a hook in MC?).


>


> Le mer. 11 mars 2020 à 23:58, Bert Freudenberg <





> bert@





> > a


> écrit :


>


>> I'd suggest to get OpenGL working outside of Croquet first:


>>


>> http://www.squeaksource.com/CroquetGL.html [http://www.squeaksource.com/CroquetGL.html]


>>


>> Step 1: Verify this works in 32 bits. (assuming you are doing this on


>> Linux, you can run 32 bit Squeak side-by-side with the 64 bit one)


>> Step 2: Make it work in 64 bits.


>>


>> The second step requires that you understand how FFI works, and how it


>> handles e.g. pointers and integer sizes.


>> I am assuming we do have a working 64 bit FFI, at least for x86_64


>> machines.


>>


>> E.g. the OGLUnix>>glExtGetProcAddress: method returns a pointer. On a 32


>> bit system, that fits into a 'ulong' which is 32 bits. On a 64 bit


>> system,


>> a pointer is 64 bits wide so it would not fit into a 32 bit word. Now I


>> don't know how many bits 'ulong' has in our 64 bit FFI, but that


>> declaration may have to change. Etc. pp.


>>


>> If you have questions about FFI then those are best directed at the


>> vm-dev


>> list since it is dealing with VM-level interfaces. CC'ing, please follow


>> up


>> there.


>>


>> - Bert -


>>


>> On Wed, Mar 11, 2020 at 2:54 PM gettimothy via Squeak-dev <


>>





> squeak-dev at .squeakfoundation





>> wrote:


>>


>>> Okey dokey,


>>>


>>> Poking along, there is a stray glyph in OGLUnix openGLLibraryName after


>>>


>>> openGLLibraryName


>>> ^Smalltalk osVersion = 'linux'


>>> ifTrue: ['libGL.so.1']


>>> ifFalse: ['GL']


>>>


>>> I removed it in my install and got past that error.


>>>


>>> Working exclusively with Croquet(Master)...


>>>


>>>


>>>


>>> My next error is in OGLUnixX11LE(OpenGL)>>glMatrixMode:


>>>


>>> glMatrixMode: mode


>>> "This method was automatically generated."


>>> "void glMatrixMode(GLenum mode);"


>>>


> <apicall: void 'glMatrixMode' (ulong) module: '#openGLLibraryName'>


>>> ^self externalCallFailed


>>>


>>> The


> <apicall:...>


>  fails


>>>


>>> How to think about this?


>>>


>>> Is Croquet behind OpenGL latest?


>>> Would teaching myself OpenGL programming be of use to the Croquet


>>> project?


>>>


>>> cheers,


>>>


>>> tty


>>>


>>>


>>>


>>

















--


Sent from: http://forum.world.st/Squeak-VM-f104410.html [http://forum.world.st/Squeak-VM-f104410.html]







--

_,,,^..^,,,_

best, Eliot


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20200609/128abbca/attachment-0001.html>


More information about the Vm-dev mailing list