[Vm-dev] [squeak-dev] 64 bit FFI (was: porting Croquet to Squeak6.0 alpha...)

Tue Jun 9 12:10:27 UTC 2020

Hi Nicolas, hi Eliot.

> If you want a type that can hold a pointer on both 32 and 64bits arch, then it's doable with a FFI type alias

You can just make an alias to a void*. It will have the right size. The only drawback would be that you might get an issue with argument coercion.

#(nil 'void*')

Your "type alias struct" will wrap around ExternalData<void*>. Why bother with ulong/ulonglong. Only if you want to work with integers in the image, you might want to use ulong or ulonglong depending on the #wordSize.

I think I will add 'word' as a type name in ExternalType's class-side "type constants" in Squeak FFI. Then you can use it in FFI call defs and struct-field defs:

ExternalType class >> #word
"Use this atomic-type alias if the external library works with opaque handles that are different on 32bit and 64bit. On a platform change, all FFI calls will be re-compiled and this method is called again by the parser."
^ FFIPlatformDescription current wordSize = 4
ifTrue: [self unsignedLong]
ifFalse: [self unsignedLongLong]

... maybe parts in the FFI plugin are faster with ulong/ulonglong compared to void* compared to type alias to void*... I don't know.

Best,
Marcel
Am 30.05.2020 18:00:33 schrieb Marcel Taeumel <marcel.taeumel at hpi.de>:
So, if the atomic types and structure types are initialized, they make use of ExternalType class >> #pointerSpec to store the size of the pointer in the #headerWord, which is the first element of an external type's 'compiledSpec'. For atomic types, this happens when the FFI package is loaded.

For structure types, this happens when a an external structure is mentioned for the first time in an FFI call method or in during compilation of another external structure's fields. See ExternalType class >> #newTypeNamed:force:. Squeak's Smalltalk parser or the compile logic in ExternalStructure invokes that type creation respectively..

Currently, if a platform change at system startup is detected, all atomic types and structure types are updated --- which implies a call to #pointerSpec in ExternalType class >> #initializeStructureTypes.

Now, what could be the situation where a struct-pointer type had the wrong pointer size? During development, your platform does not magically change the #wordSize. When loading code, methods in the external struct might be wrong (bc. Monticello does not trigger that field re-definition) BUT any external struct-pointer type that is already in the image must have the right size. And for new external struct-pointer types, it should work as explained above.

So, I think we should make the following changes:
- Remove ExternalType >> #asPointerType: (! note the colon !)
- Remove ExternalType >> #pointerSize:
- Change ExternalType >> #pointerSize to just read the compiledSpec
- Update all sends to #asPointerType: to #asPointerType and assume that the current pointer-type instances have the right size, as described above.

Best,
Marcel

Am 30.05.2020 15:59:31 schrieb Marcel Taeumel <marcel.taeumel at hpi.de>:
> But if it is obsolete now, nuke it.

I cannot tell. That information is stored in the external type's compiledSpec to be used from within the FFI plugin I suppose.

Best,
Marcel
Am 29.05.2020 00:15:25 schrieb Eliot Miranda <eliot.miranda at gmail.com>:
On Thu, May 28, 2020 at 11:33 AM Nicolas Cellier <>
nicolas.cellier.aka.nice at gmail.com> wrote:

>
> Hi Marcel,
> no idea...
> Look at timestamp:
> eem 2/16/2016 12:42 · accessing · 2 implementors · only in change set
> FFI-Kernel-EstebanLorenzano.45 ·
> So presumably Eliot added that.
>

Perhaps. I don't remember. But if it is obsolete now, nuke it.

>
> Le jeu. 28 mai 2020 à 17:00, marcel.taeumel a
> écrit :
>
>>
>> Hi Nicolas,
>>
>> what is the role of ExternalStructure class >> #pointerSize then?
>>
>> Best,
>> Marcel
>>
>>
>> Nicolas Cellier wrote
>> > Hi all,
>> > beware of FFI 'long' and 'ulong'.
>> > They are currently meaning (32 bits) int and unsigned int!
>> > This dates back from the 80s when there were still odd OS with segmented
>> > memory and 16-bits int.
>> > long was thought as a safe int32_t at that time... and still means just
>> > that in current FFI.
>> >
>> > I'd like to replace them with 'int' and 'uint', but we need to ensure
>> > backward compatibility etc...
>> >
>> > FFI has ExternalType signedLongLong or just 'longlong' and ExternalType
>> > unsignedLongLong or 'ulonglong' which are 64 bits on all architectures.
>> > See implementors of #initializeAtomicTypes #initializeFFIConstants
>> >
>> > If you want a type that can hold a pointer on both 32 and 64bits arch,
>> > then
>> > it's doable with a FFI type alias
>> > For example I have things like this in HDF5 module
>> > Notice the single #(fieldName fieldType) which defines an alias, while
>> > ordinary FFI struct gets an array of #(fieldName fieldType)
>> > (and replace Size_t with Intptr_t if the purpose is hodling a pointer,
>> > otherwise the ayatollahs of C standard will lash you)
>> >
>> > ExternalStructure subclass: #'Size_t'
>> > instanceVariableNames: ''
>> > classVariableNames: ''
>> > poolDictionaries: ''
>> > category: 'HDF5-External-atomic'
>> >
>> > Size_t class >>fields
>> > "Size_t defineFields"
>> > ^Smalltalk wordSize = 4
>> > ifTrue: [#( value 'ulong')]
>> > ifFalse: [#( value 'ulonglong')]
>> >
>> > However the "fields" must be re-generated when platform changes...
>> > I tried to automate that sometimes ago, see:
>> >
>> > ExternalStructure class>>install
>> > "Resuming the image on another architecture may require a
>> > re-compilation of structure layout."
>> > | newPlatform |
>> > newPlatform := Smalltalk platformName.
>> > PreviousPlatform = newPlatform
>> > ifFalse:
>> > [self recompileStructures.
>> > PreviousPlatform := newPlatform]
>> >
>> > It works if you switch from Win64 to MacOS64 for example...
>> > However, this mechanism is not used yet at time of code loading, so
>> > importing code generated on a different platform won't automatically
>> > update
>> > the structures (or alias) - the FFI package has no mechanism for that
>> > (maybe we could add a hook in MC?).
>> >
>> > Le mer. 11 mars 2020 à 23:58, Bert Freudenberg <
>>
>> > bert@
>>
>> > > a
>> > écrit :
>> >
>> >> I'd suggest to get OpenGL working outside of Croquet first:
>> >>
>> >> http://www.squeaksource.com/CroquetGL.html
>> >>
>> >> Step 1: Verify this works in 32 bits. (assuming you are doing this on
>> >> Linux, you can run 32 bit Squeak side-by-side with the 64 bit one)
>> >> Step 2: Make it work in 64 bits.
>> >>
>> >> The second step requires that you understand how FFI works, and how it
>> >> handles e.g. pointers and integer sizes.
>> >> I am assuming we do have a working 64 bit FFI, at least for x86_64
>> >> machines.
>> >>
>> >> E.g. the OGLUnix>>glExtGetProcAddress: method returns a pointer. On a
>> 32
>> >> bit system, that fits into a 'ulong' which is 32 bits. On a 64 bit
>> >> system,
>> >> a pointer is 64 bits wide so it would not fit into a 32 bit word. Now I
>> >> don't know how many bits 'ulong' has in our 64 bit FFI, but that
>> >> declaration may have to change. Etc. pp.
>> >>
>> >> If you have questions about FFI then those are best directed at the
>> >> vm-dev
>> >> list since it is dealing with VM-level interfaces. CC'ing, please
>> follow
>> >> up
>> >> there.
>> >>
>> >> - Bert -
>> >>
>> >> On Wed, Mar 11, 2020 at 2:54 PM gettimothy via Squeak-dev <>
>> >>
>>
>> > squeak-dev at .squeakfoundation
>>
>> >> wrote:
>> >>
>> >>> Okey dokey,
>> >>>
>> >>> Poking along, there is a stray glyph in OGLUnix openGLLibraryName
>> after
>> >>>
>> >>> openGLLibraryName
>> >>> ^Smalltalk osVersion = 'linux'
>> >>> ifTrue: ['libGL.so.1']
>> >>> ifFalse: ['GL']
>> >>>
>> >>> I removed it in my install and got past that error.
>> >>>
>> >>> Working exclusively with Croquet(Master)...
>> >>>
>> >>>
>> >>>
>> >>> My next error is in OGLUnixX11LE(OpenGL)>>glMatrixMode:
>> >>>
>> >>> glMatrixMode: mode
>> >>> "This method was automatically generated."
>> >>> "void glMatrixMode(GLenum mode);"
>> >>>
>> >
>> >>> ^self externalCallFailed
>> >>>
>> >>> The
>> >
>> > fails
>> >>>
>> >>> How to think about this?
>> >>>
>> >>> Is Croquet behind OpenGL latest?
>> >>> Would teaching myself OpenGL programming be of use to the Croquet
>> >>> project?
>> >>>
>> >>> cheers,
>> >>>
>> >>> tty
>> >>>
>> >>>
>> >>>
>> >>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://forum.world.st/Squeak-VM-f104410.html
>>
>

--
_,,,^..^,,,_
best, Eliot

On Thu, May 28, 2020 at 11:33 AM Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com [mailto:nicolas.cellier.aka.nice at gmail.com]> wrote:

Hi Marcel,
no idea...
Look at timestamp:
eem 2/16/2016 12:42 · accessing · 2 implementors · only in change set FFI-Kernel-EstebanLorenzano.45  ·
So presumably Eliot added that.

Perhaps.  I don't remember.  But if it is obsolete now, nuke it.

Le jeu. 28 mai 2020 à 17:00, marcel.taeumel <Marcel.Taeumel at hpi.de [mailto:Marcel.Taeumel at hpi.de]> a écrit :

Hi Nicolas,

what is the role of ExternalStructure class >> #pointerSize then?

Best,

Marcel

Nicolas Cellier wrote

> Hi all,

> beware of FFI 'long' and 'ulong'.

> They are currently meaning (32 bits) int and unsigned int!

> This dates back from the 80s when there were still odd OS with segmented

> memory and 16-bits int.

> long was thought as a safe int32_t at that time... and still means just

> that in current FFI.

>

> I'd like to replace them with 'int' and 'uint', but we need to ensure

> backward compatibility etc...

>

> FFI has ExternalType signedLongLong or just 'longlong' and ExternalType

> unsignedLongLong or 'ulonglong' which are 64 bits on all architectures.

> See implementors of #initializeAtomicTypes #initializeFFIConstants

>

> If you want a type that can hold a pointer on both 32 and 64bits arch,

> then

> it's doable with a FFI type alias

> For example I have things like this in HDF5 module

> Notice the single #(fieldName fieldType) which defines an alias, while

> ordinary FFI struct gets an array of #(fieldName fieldType)

> (and replace Size_t with Intptr_t if the purpose is hodling a pointer,

> otherwise the ayatollahs of C standard will lash you)

>

> ExternalStructure subclass: #'Size_t'

>     instanceVariableNames: ''

>     classVariableNames: ''

>     poolDictionaries: ''

>     category: 'HDF5-External-atomic'

>

> Size_t class >>fields

>     "Size_t defineFields"

>     ^Smalltalk wordSize = 4

>         ifTrue: [#( value 'ulong')]

>         ifFalse: [#( value 'ulonglong')]

>

> However the "fields" must be re-generated when platform changes...

> I tried to automate that sometimes ago, see:

>

> ExternalStructure class>>install

>      "Resuming the image on another architecture may require a

> re-compilation of structure layout."

>      | newPlatform |

>      newPlatform := Smalltalk platformName.

>      PreviousPlatform = newPlatform

>          ifFalse:

>              [self recompileStructures.

>             PreviousPlatform := newPlatform]

>

> It works if you switch from Win64 to MacOS64 for example...

> However, this mechanism is not used yet at time of code loading, so

> importing code generated on a different platform won't automatically

> update

> the structures (or alias) - the FFI package has no mechanism for that

> (maybe we could add a hook in MC?).

>

> Le mer. 11 mars 2020 à 23:58, Bert Freudenberg <

> bert@

> > a

> écrit :

>

>> I'd suggest to get OpenGL working outside of Croquet first:

>>

>> http://www.squeaksource.com/CroquetGL.html [http://www.squeaksource.com/CroquetGL.html]

>>

>> Step 1: Verify this works in 32 bits. (assuming you are doing this on

>> Linux, you can run 32 bit Squeak side-by-side with the 64 bit one)

>> Step 2: Make it work in 64 bits.

>>

>> The second step requires that you understand how FFI works, and how it

>> handles e.g. pointers and integer sizes.

>> I am assuming we do have a working 64 bit FFI, at least for x86_64

>> machines.

>>

>> E.g. the OGLUnix>>glExtGetProcAddress: method returns a pointer. On a 32

>> bit system, that fits into a 'ulong' which is 32 bits. On a 64 bit

>> system,

>> a pointer is 64 bits wide so it would not fit into a 32 bit word. Now I

>> don't know how many bits 'ulong' has in our 64 bit FFI, but that

>> declaration may have to change. Etc. pp.

>>

>> If you have questions about FFI then those are best directed at the

>> vm-dev

>> list since it is dealing with VM-level interfaces. CC'ing, please follow

>> up

>> there.

>>

>> - Bert -

>>

>> On Wed, Mar 11, 2020 at 2:54 PM gettimothy via Squeak-dev <

>>

> squeak-dev at .squeakfoundation

>> wrote:

>>

>>> Okey dokey,

>>>

>>> Poking along, there is a stray glyph in OGLUnix openGLLibraryName after

>>>

>>> openGLLibraryName

>>> ^Smalltalk osVersion = 'linux'

>>> ifTrue: ['libGL.so.1']

>>> ifFalse: ['GL']

>>>

>>> I removed it in my install and got past that error.

>>>

>>> Working exclusively with Croquet(Master)...

>>>

>>>

>>>

>>> My next error is in OGLUnixX11LE(OpenGL)>>glMatrixMode:

>>>

>>> glMatrixMode: mode

>>> "This method was automatically generated."

>>> "void glMatrixMode(GLenum mode);"

>>>

> <apicall: void 'glMatrixMode' (ulong) module: '#openGLLibraryName'>

>>> ^self externalCallFailed

>>>

>>> The

> <apicall:...>

>  fails

>>>

>>> How to think about this?

>>>

>>> Is Croquet behind OpenGL latest?

>>> Would teaching myself OpenGL programming be of use to the Croquet

>>> project?

>>>

>>> cheers,

>>>

>>> tty

>>>

>>>

>>>

>>

--

Sent from: http://forum.world.st/Squeak-VM-f104410.html [http://forum.world.st/Squeak-VM-f104410.html]

--

_,,,^..^,,,_

best, Eliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20200609/c08f2713/attachment-0001.html>