[Vm-dev] [squeak-dev] 64 bit FFI (was: porting Croquet to Squeak6.0 alpha...)

Eliot Miranda eliot.miranda at gmail.com
Tue Jun 9 16:33:21 UTC 2020 

> On Jun 9, 2020, at 5:10 AM, Marcel Taeumel <marcel.taeumel at hpi.de> wrote:
> 
> 
> Hi Nicolas, hi Eliot.
> 
> > If you want a type that can hold a pointer on both 32 and 64bits arch, then it's doable with a FFI type alias
> 
> You can just make an alias to a void*. It will have the right size. The only drawback would be that you might get an issue with argument coercion.
> 
> #(nil 'void*')
> 
> Your "type alias struct" will wrap around ExternalData<void*>. Why bother with ulong/ulonglong. Only if you want to work with integers in the image, you might want to use ulong or ulonglong depending on the #wordSize.
> 
> I think I will add 'word' as a type name in ExternalType's class-side "type constants" in Squeak FFI. Then you can use it in FFI call defs and struct-field defs:
> 
> ExternalType class >> #word
> 	"Use this atomic-type alias if the external library works with opaque handles that are different on 32bit and 64bit. On a platform change, all FFI calls will be re-compiled and this method is called again by the parser."
> 	
> 	^ FFIPlatformDescription current wordSize = 4
> 		ifTrue: [self unsignedLong]
> 		ifFalse: [self unsignedLongLong]
> 
> ... maybe parts in the FFI plugin are faster with ulong/ulonglong compared to void* compared to type alias to void*... I don't know.

It might be better to use a C name such as intptr_t or uintptr_t.

> 
> Best,
> Marcel
>> Am 30.05.2020 18:00:33 schrieb Marcel Taeumel <marcel.taeumel at hpi.de>:
>> 
>> So, if the atomic types and structure types are initialized, they make use of ExternalType class >> #pointerSpec to store the size of the pointer in the #headerWord, which is the first element of an external type's 'compiledSpec'. For atomic types, this happens when the FFI package is loaded.
>> 
>> For structure types, this happens when a an external structure is mentioned for the first time in an FFI call method or in during compilation of another external structure's fields. See ExternalType class >> #newTypeNamed:force:. Squeak's Smalltalk parser or the compile logic in ExternalStructure invokes that type creation respectively..
>> 
>> Currently, if a platform change at system startup is detected, all atomic types and structure types are updated --- which implies a call to #pointerSpec in ExternalType class >> #initializeStructureTypes.
>> 
>> Now, what could be the situation where a struct-pointer type had the wrong pointer size? During development, your platform does not magically change the #wordSize. When loading code, methods in the external struct might be wrong (bc. Monticello does not trigger that field re-definition) BUT any external struct-pointer type that is already in the image must have the right size. And for new external struct-pointer types, it should work as explained above.
>> 
>> So, I think we should make the following changes:
>> - Remove ExternalType >> #asPointerType: (! note the colon !)
>> - Remove ExternalType >> #pointerSize:
>> - Change ExternalType >> #pointerSize to just read the compiledSpec
>> - Update all sends to #asPointerType: to #asPointerType and assume that the current pointer-type instances have the right size, as described above.
>> 
>> Best,
>> Marcel
>> 
>>> Am 30.05.2020 15:59:31 schrieb Marcel Taeumel <marcel.taeumel at hpi.de>:
>>> 
>>> > But if it is obsolete now, nuke it.
>>> 
>>> I cannot tell. That information is stored in the external type's compiledSpec to be used from within the FFI plugin I suppose.
>>> 
>>> Best,
>>> Marcel
>>>> Am 29.05.2020 00:15:25 schrieb Eliot Miranda <eliot.miranda at gmail.com>:
>>>> 
>>>> On Thu, May 28, 2020 at 11:33 AM Nicolas Cellier <>
>>>> nicolas.cellier.aka.nice at gmail.com> wrote: 
>>>> 
>>>> > 
>>>> > Hi Marcel, 
>>>> > no idea... 
>>>> > Look at timestamp: 
>>>> > eem 2/16/2016 12:42 · accessing · 2 implementors · only in change set 
>>>> > FFI-Kernel-EstebanLorenzano.45 · 
>>>> > So presumably Eliot added that. 
>>>> > 
>>>> 
>>>> Perhaps. I don't remember. But if it is obsolete now, nuke it. 
>>>> 
>>>> 
>>>> > 
>>>> > Le jeu. 28 mai 2020 à 17:00, marcel.taeumel a 
>>>> > écrit : 
>>>> > 
>>>> >> 
>>>> >> Hi Nicolas, 
>>>> >> 
>>>> >> what is the role of ExternalStructure class >> #pointerSize then? 
>>>> >> 
>>>> >> Best, 
>>>> >> Marcel 
>>>> >> 
>>>> >> 
>>>> >> Nicolas Cellier wrote 
>>>> >> > Hi all, 
>>>> >> > beware of FFI 'long' and 'ulong'. 
>>>> >> > They are currently meaning (32 bits) int and unsigned int! 
>>>> >> > This dates back from the 80s when there were still odd OS with segmented 
>>>> >> > memory and 16-bits int. 
>>>> >> > long was thought as a safe int32_t at that time... and still means just 
>>>> >> > that in current FFI. 
>>>> >> > 
>>>> >> > I'd like to replace them with 'int' and 'uint', but we need to ensure 
>>>> >> > backward compatibility etc... 
>>>> >> > 
>>>> >> > FFI has ExternalType signedLongLong or just 'longlong' and ExternalType 
>>>> >> > unsignedLongLong or 'ulonglong' which are 64 bits on all architectures. 
>>>> >> > See implementors of #initializeAtomicTypes #initializeFFIConstants 
>>>> >> > 
>>>> >> > If you want a type that can hold a pointer on both 32 and 64bits arch, 
>>>> >> > then 
>>>> >> > it's doable with a FFI type alias 
>>>> >> > For example I have things like this in HDF5 module 
>>>> >> > Notice the single #(fieldName fieldType) which defines an alias, while 
>>>> >> > ordinary FFI struct gets an array of #(fieldName fieldType) 
>>>> >> > (and replace Size_t with Intptr_t if the purpose is hodling a pointer, 
>>>> >> > otherwise the ayatollahs of C standard will lash you) 
>>>> >> > 
>>>> >> > ExternalStructure subclass: #'Size_t' 
>>>> >> > instanceVariableNames: '' 
>>>> >> > classVariableNames: '' 
>>>> >> > poolDictionaries: '' 
>>>> >> > category: 'HDF5-External-atomic' 
>>>> >> > 
>>>> >> > Size_t class >>fields 
>>>> >> > "Size_t defineFields" 
>>>> >> > ^Smalltalk wordSize = 4 
>>>> >> > ifTrue: [#( value 'ulong')] 
>>>> >> > ifFalse: [#( value 'ulonglong')] 
>>>> >> > 
>>>> >> > However the "fields" must be re-generated when platform changes... 
>>>> >> > I tried to automate that sometimes ago, see: 
>>>> >> > 
>>>> >> > ExternalStructure class>>install 
>>>> >> > "Resuming the image on another architecture may require a 
>>>> >> > re-compilation of structure layout." 
>>>> >> > | newPlatform | 
>>>> >> > newPlatform := Smalltalk platformName. 
>>>> >> > PreviousPlatform = newPlatform 
>>>> >> > ifFalse: 
>>>> >> > [self recompileStructures. 
>>>> >> > PreviousPlatform := newPlatform] 
>>>> >> > 
>>>> >> > It works if you switch from Win64 to MacOS64 for example... 
>>>> >> > However, this mechanism is not used yet at time of code loading, so 
>>>> >> > importing code generated on a different platform won't automatically 
>>>> >> > update 
>>>> >> > the structures (or alias) - the FFI package has no mechanism for that 
>>>> >> > (maybe we could add a hook in MC?). 
>>>> >> > 
>>>> >> > Le mer. 11 mars 2020 à 23:58, Bert Freudenberg < 
>>>> >> 
>>>> >> > bert@ 
>>>> >> 
>>>> >> > > a 
>>>> >> > écrit : 
>>>> >> > 
>>>> >> >> I'd suggest to get OpenGL working outside of Croquet first: 
>>>> >> >> 
>>>> >> >> http://www.squeaksource.com/CroquetGL.html 
>>>> >> >> 
>>>> >> >> Step 1: Verify this works in 32 bits. (assuming you are doing this on 
>>>> >> >> Linux, you can run 32 bit Squeak side-by-side with the 64 bit one) 
>>>> >> >> Step 2: Make it work in 64 bits. 
>>>> >> >> 
>>>> >> >> The second step requires that you understand how FFI works, and how it 
>>>> >> >> handles e.g. pointers and integer sizes. 
>>>> >> >> I am assuming we do have a working 64 bit FFI, at least for x86_64 
>>>> >> >> machines. 
>>>> >> >> 
>>>> >> >> E.g. the OGLUnix>>glExtGetProcAddress: method returns a pointer. On a 
>>>> >> 32 
>>>> >> >> bit system, that fits into a 'ulong' which is 32 bits. On a 64 bit 
>>>> >> >> system, 
>>>> >> >> a pointer is 64 bits wide so it would not fit into a 32 bit word. Now I 
>>>> >> >> don't know how many bits 'ulong' has in our 64 bit FFI, but that 
>>>> >> >> declaration may have to change. Etc. pp. 
>>>> >> >> 
>>>> >> >> If you have questions about FFI then those are best directed at the 
>>>> >> >> vm-dev 
>>>> >> >> list since it is dealing with VM-level interfaces. CC'ing, please 
>>>> >> follow 
>>>> >> >> up 
>>>> >> >> there. 
>>>> >> >> 
>>>> >> >> - Bert - 
>>>> >> >> 
>>>> >> >> On Wed, Mar 11, 2020 at 2:54 PM gettimothy via Squeak-dev <>
>>>> >> >> 
>>>> >> 
>>>> >> > squeak-dev at .squeakfoundation 
>>>> >> 
>>>> >> >> wrote: 
>>>> >> >> 
>>>> >> >>> Okey dokey, 
>>>> >> >>> 
>>>> >> >>> Poking along, there is a stray glyph in OGLUnix openGLLibraryName 
>>>> >> after 
>>>> >> >>> 
>>>> >> >>> openGLLibraryName 
>>>> >> >>> ^Smalltalk osVersion = 'linux' 
>>>> >> >>> ifTrue: ['libGL.so.1'] 
>>>> >> >>> ifFalse: ['GL'] 
>>>> >> >>> 
>>>> >> >>> I removed it in my install and got past that error. 
>>>> >> >>> 
>>>> >> >>> Working exclusively with Croquet(Master)... 
>>>> >> >>> 
>>>> >> >>> 
>>>> >> >>> 
>>>> >> >>> My next error is in OGLUnixX11LE(OpenGL)>>glMatrixMode: 
>>>> >> >>> 
>>>> >> >>> glMatrixMode: mode 
>>>> >> >>> "This method was automatically generated." 
>>>> >> >>> "void glMatrixMode(GLenum mode);" 
>>>> >> >>> 
>>>> >> > 
>>>> >> >>> ^self externalCallFailed 
>>>> >> >>> 
>>>> >> >>> The 
>>>> >> > 
>>>> >> > fails 
>>>> >> >>> 
>>>> >> >>> How to think about this? 
>>>> >> >>> 
>>>> >> >>> Is Croquet behind OpenGL latest? 
>>>> >> >>> Would teaching myself OpenGL programming be of use to the Croquet 
>>>> >> >>> project? 
>>>> >> >>> 
>>>> >> >>> cheers, 
>>>> >> >>> 
>>>> >> >>> tty 
>>>> >> >>> 
>>>> >> >>> 
>>>> >> >>> 
>>>> >> >> 
>>>> >> 
>>>> >> 
>>>> >> 
>>>> >> 
>>>> >> 
>>>> >> -- 
>>>> >> Sent from: http://forum.world.st/Squeak-VM-f104410.html 
>>>> >> 
>>>> > 
>>>> 
>>>> -- 
>>>> _,,,^..^,,,_ 
>>>> best, Eliot 
>>>> 
>>>> 
>>>> On Thu, May 28, 2020 at 11:33 AM Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com> wrote:
>>>>>  
>>>>> Hi Marcel,
>>>>> no idea...
>>>>> Look at timestamp:
>>>>> eem 2/16/2016 12:42 · accessing · 2 implementors · only in change set FFI-Kernel-EstebanLorenzano.45  ·
>>>>> So presumably Eliot added that.
>>>> 
>>>> Perhaps.  I don't remember.  But if it is obsolete now, nuke it.
>>>>  
>>>>> 
>>>>> Le jeu. 28 mai 2020 à 17:00, marcel.taeumel <Marcel.Taeumel at hpi.de> a écrit :
>>>>>>  
>>>>>> 
>>>>>> Hi Nicolas,
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> what is the role of ExternalStructure class >> #pointerSize then?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Best,
>>>>>> 
>>>>>> Marcel
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Nicolas Cellier wrote
>>>>>> 
>>>>>> > Hi all,
>>>>>> 
>>>>>> > beware of FFI 'long' and 'ulong'.
>>>>>> 
>>>>>> > They are currently meaning (32 bits) int and unsigned int!
>>>>>> 
>>>>>> > This dates back from the 80s when there were still odd OS with segmented
>>>>>> 
>>>>>> > memory and 16-bits int.
>>>>>> 
>>>>>> > long was thought as a safe int32_t at that time... and still means just
>>>>>> 
>>>>>> > that in current FFI.
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> > I'd like to replace them with 'int' and 'uint', but we need to ensure
>>>>>> 
>>>>>> > backward compatibility etc...
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> > FFI has ExternalType signedLongLong or just 'longlong' and ExternalType
>>>>>> 
>>>>>> > unsignedLongLong or 'ulonglong' which are 64 bits on all architectures.
>>>>>> 
>>>>>> > See implementors of #initializeAtomicTypes #initializeFFIConstants
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> > If you want a type that can hold a pointer on both 32 and 64bits arch,
>>>>>> 
>>>>>> > then
>>>>>> 
>>>>>> > it's doable with a FFI type alias
>>>>>> 
>>>>>> > For example I have things like this in HDF5 module
>>>>>> 
>>>>>> > Notice the single #(fieldName fieldType) which defines an alias, while
>>>>>> 
>>>>>> > ordinary FFI struct gets an array of #(fieldName fieldType)
>>>>>> 
>>>>>> > (and replace Size_t with Intptr_t if the purpose is hodling a pointer,
>>>>>> 
>>>>>> > otherwise the ayatollahs of C standard will lash you)
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> > ExternalStructure subclass: #'Size_t'
>>>>>> 
>>>>>> >     instanceVariableNames: ''
>>>>>> 
>>>>>> >     classVariableNames: ''
>>>>>> 
>>>>>> >     poolDictionaries: ''
>>>>>> 
>>>>>> >     category: 'HDF5-External-atomic'
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> > Size_t class >>fields
>>>>>> 
>>>>>> >     "Size_t defineFields"
>>>>>> 
>>>>>> >     ^Smalltalk wordSize = 4
>>>>>> 
>>>>>> >         ifTrue: [#( value 'ulong')]
>>>>>> 
>>>>>> >         ifFalse: [#( value 'ulonglong')]
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> > However the "fields" must be re-generated when platform changes...
>>>>>> 
>>>>>> > I tried to automate that sometimes ago, see:
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> > ExternalStructure class>>install
>>>>>> 
>>>>>> >      "Resuming the image on another architecture may require a
>>>>>> 
>>>>>> > re-compilation of structure layout."
>>>>>> 
>>>>>> >      | newPlatform |
>>>>>> 
>>>>>> >      newPlatform := Smalltalk platformName.
>>>>>> 
>>>>>> >      PreviousPlatform = newPlatform
>>>>>> 
>>>>>> >          ifFalse:
>>>>>> 
>>>>>> >              [self recompileStructures.
>>>>>> 
>>>>>> >             PreviousPlatform := newPlatform]
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> > It works if you switch from Win64 to MacOS64 for example...
>>>>>> 
>>>>>> > However, this mechanism is not used yet at time of code loading, so
>>>>>> 
>>>>>> > importing code generated on a different platform won't automatically
>>>>>> 
>>>>>> > update
>>>>>> 
>>>>>> > the structures (or alias) - the FFI package has no mechanism for that
>>>>>> 
>>>>>> > (maybe we could add a hook in MC?).
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> > Le mer. 11 mars 2020 à 23:58, Bert Freudenberg <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> > bert@
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> > > a
>>>>>> 
>>>>>> > écrit :
>>>>>> 
>>>>>> > 
>>>>>> 
>>>>>> >> I'd suggest to get OpenGL working outside of Croquet first:
>>>>>> 
>>>>>> >>
>>>>>> 
>>>>>> >> http://www.squeaksource.com/CroquetGL.html
>>>>>> 
>>>>>> >>
>>>>>> 
>>>>>> >> Step 1: Verify this works in 32 bits. (assuming you are doing this on
>>>>>> 
>>>>>> >> Linux, you can run 32 bit Squeak side-by-side with the 64 bit one)
>>>>>> 
>>>>>> >> Step 2: Make it work in 64 bits.
>>>>>> 
>>>>>> >>
>>>>>> 
>>>>>> >> The second step requires that you understand how FFI works, and how it
>>>>>> 
>>>>>> >> handles e.g. pointers and integer sizes.
>>>>>> 
>>>>>> >> I am assuming we do have a working 64 bit FFI, at least for x86_64
>>>>>> 
>>>>>> >> machines.
>>>>>> 
>>>>>> >>
>>>>>> 
>>>>>> >> E.g. the OGLUnix>>glExtGetProcAddress: method returns a pointer. On a 32
>>>>>> 
>>>>>> >> bit system, that fits into a 'ulong' which is 32 bits. On a 64 bit
>>>>>> 
>>>>>> >> system,
>>>>>> 
>>>>>> >> a pointer is 64 bits wide so it would not fit into a 32 bit word. Now I
>>>>>> 
>>>>>> >> don't know how many bits 'ulong' has in our 64 bit FFI, but that
>>>>>> 
>>>>>> >> declaration may have to change. Etc. pp.
>>>>>> 
>>>>>> >>
>>>>>> 
>>>>>> >> If you have questions about FFI then those are best directed at the
>>>>>> 
>>>>>> >> vm-dev
>>>>>> 
>>>>>> >> list since it is dealing with VM-level interfaces. CC'ing, please follow
>>>>>> 
>>>>>> >> up
>>>>>> 
>>>>>> >> there.
>>>>>> 
>>>>>> >>
>>>>>> 
>>>>>> >> - Bert -
>>>>>> 
>>>>>> >>
>>>>>> 
>>>>>> >> On Wed, Mar 11, 2020 at 2:54 PM gettimothy via Squeak-dev <
>>>>>> 
>>>>>> >> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> > squeak-dev at .squeakfoundation
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> >> wrote:
>>>>>> 
>>>>>> >>
>>>>>> 
>>>>>> >>> Okey dokey,
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> Poking along, there is a stray glyph in OGLUnix openGLLibraryName after
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> openGLLibraryName
>>>>>> 
>>>>>> >>> ^Smalltalk osVersion = 'linux'
>>>>>> 
>>>>>> >>> ifTrue: ['libGL.so.1']
>>>>>> 
>>>>>> >>> ifFalse: ['GL']
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> I removed it in my install and got past that error.
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> Working exclusively with Croquet(Master)...
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> My next error is in OGLUnixX11LE(OpenGL)>>glMatrixMode:
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> glMatrixMode: mode
>>>>>> 
>>>>>> >>> "This method was automatically generated."
>>>>>> 
>>>>>> >>> "void glMatrixMode(GLenum mode);"
>>>>>> 
>>>>>> >>> 
>>>>>> 
>>>>>> > <apicall: void 'glMatrixMode' (ulong) module: '#openGLLibraryName'>
>>>>>> 
>>>>>> >>> ^self externalCallFailed
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> The 
>>>>>> 
>>>>>> > <apicall:...>
>>>>>> 
>>>>>> >  fails
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> How to think about this?
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> Is Croquet behind OpenGL latest?
>>>>>> 
>>>>>> >>> Would teaching myself OpenGL programming be of use to the Croquet
>>>>>> 
>>>>>> >>> project?
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> cheers,
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>> tty
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>>
>>>>>> 
>>>>>> >>
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> 
>>>>>> Sent from: http://forum.world.st/Squeak-VM-f104410.html
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> _,,,^..^,,,_
>>>> best, Eliot
>>>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20200609/63490fc2/attachment-0001.html>

More information about the Vm-dev mailing list