[Vm-dev] nested structures with FFI?
Esteban Lorenzano
estebanlm at gmail.com
Fri Aug 21 10:47:47 UTC 2015
> On 21 Aug 2015, at 12:42, Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com> wrote:
>
>
>
> 2015-08-21 11:53 GMT+02:00 Esteban Lorenzano <estebanlm at gmail.com <mailto:estebanlm at gmail.com>>:
>
>
>> On 21 Aug 2015, at 11:45, Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com <mailto:nicolas.cellier.aka.nice at gmail.com>> wrote:
>>
>>
>>
>> 2015-08-21 5:57 GMT+02:00 Esteban Lorenzano <estebanlm at gmail.com <mailto:estebanlm at gmail.com>>:
>>
>>
>>> On 21 Aug 2015, at 04:02, Esteban Lorenzano <estebanlm at gmail.com <mailto:estebanlm at gmail.com>> wrote:
>>>
>>>>
>>>> On 20 Aug 2015, at 22:47, Esteban Lorenzano <estebanlm at gmail.com <mailto:estebanlm at gmail.com>> wrote:
>>>>
>>>>>
>>>>> On 20 Aug 2015, at 21:48, Max Leske <maxleske at gmail.com <mailto:maxleske at gmail.com>> wrote:
>>>>>
>>>>>
>>>>>> On 20 Aug 2015, at 21:28, vm-dev-request at lists.squeakfoundation.org <mailto:vm-dev-request at lists.squeakfoundation.org> wrote:
>>>>>>
>>>>>> Message: 1
>>>>>> Date: Thu, 20 Aug 2015 17:51:43 +0200
>>>>>> From: Esteban Lorenzano <estebanlm at gmail.com <mailto:estebanlm at gmail.com>>
>>>>>> Subject: [Vm-dev] nested structures with FFI?
>>>>>> To: Squeak Virtual Machine Development Discussion
>>>>>> <vm-dev at lists.squeakfoundation.org <mailto:vm-dev at lists.squeakfoundation.org>>
>>>>>> Message-ID: <4F31ED4B-6246-43D6-B85B-7215D8570D28 at gmail.com <mailto:4F31ED4B-6246-43D6-B85B-7215D8570D28 at gmail.com>>
>>>>>> Content-Type: text/plain; charset=utf-8
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure.
>>>>>> For example, this tests:
>>>>>>
>>>>>> FFITestNestingStructure class>>#fieldsDesc
>>>>>> ^ #(
>>>>>> byte one;
>>>>>> FFITestStructure nested;
>>>>>> )
>>>>>>
>>>>>> FFITestStructure class>>#fieldsDesc
>>>>>> ^ #(
>>>>>> byte byte;
>>>>>> short short;
>>>>>> long long;
>>>>>> float float;
>>>>>> double double;
>>>>>> int64 int64;
>>>>>> )
>>>>>>
>>>>>> (definition is NB-compatible, so do not take the differences with regular FFI into account)
>>>>>>
>>>>>> testNestedStructure
>>>>>> | s1 |
>>>>>>
>>>>>> s1 := FFITestNestingStructure new.
>>>>>> s1 nested byte: 42.
>>>>>> self assert: s1 nested byte = 42
>>>>>>
>>>>>> testExternallyAllocatedNestedStructure
>>>>>> | s1 |
>>>>>>
>>>>>> s1 := FFITestNestingStructure externalNew.
>>>>>> self assert: s1 getHandle class = ExternalAddress.
>>>>>> s1 nested byte: 42.
>>>>>> self assert: s1 nested byte = 42
>>>>>>
>>>>>>
>>>>>> fails always because
>>>>>>
>>>>>> s1 nested byte = 0.
>>>>>>
>>>>>> (which is obviously bad).
>>>>>>
>>>>>> I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
>>>>>
>>>>> I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
>>>>>
>>>>> Not sure if this helps you in any way but I thought I’d let you know :)
>>>>
>>>> I do not think so :)
>>>> I’m quite sure this is not going to be a problem when using the ThreadedFFI backend instead ASMJIT… now my problem is with this backend, for the moment :)
>>>
>>> I’m just realised that this behaviour is correct according current ExternalStructure implementation, because:
>>>
>>> nested
>>> "This method was automatically generated"
>>> ^FFITestStructure fromHandle: (handle structAt: 2 length: 27)
>>>
>>> and
>>>
>>> ByteArray>>#structAt: byteOffset length: length
>>> "Return a structure of the given length starting at the indicated byte offset."
>>> | value |
>>> value := ByteArray new: length.
>>> 1 to: length do:[:i|
>>> value unsignedByteAt: i put: (self unsignedByteAt: byteOffset+i-1)].
>>> ^value
>>>
>>> So it is giving me always a copy…
>>> So we need a better solution (a non copy way to access)? I’m thinking on passing a “MappedByteArray” something that can work with the same original ByteArray (or ExternalAddress, is the same), to affect the referenced data… what do you think?
>>
>> So yes, introducing a "referenced handle” (who applies offsets to same memory address or byte array) makes everything work… and looks like the correct way to handle this.
>> Do you want I commit this changes to FFI package or I keep them in my own stuff?
>>
>> Esteban
>>
>>
>> Yes!
>>
>> years ago when I ported Smallapack to Squeak, that just was the missing sugar that would have enabled implementing all using Smalltalk memory...
>>
>> Instead of that, I had to copy data to external heap in order to perform certain operations.
>> For example, for extracting the imaginary part of a complex matrix interlaced representation { re[0] , im[0] , re[1] , im[1], ... } one trick is to cast a DoubleComplexMatrix * into a DoubleMatrix *, and start the copy at doubleMatrixPointer+1, with increment=2…
>
> yes, but you are talking of something more complex than what I did, I think (what I did just works for nested structures)… what you want is a functionality I also believe I will need to add anyway :)
> so I will keep playing with this and will commit when ready.
>
> cheers!
> Esteban
>
>
> If we have possibility to pass a composed object (ByteArray + byte_offset) to FFI and have it transformed in a (char *) ByteArray_start_of_data + byte_offset, then it solves all the problems.
> That's the only feature I need, the complex part in my example (casting etc...) is completely handled in image code.
yes, I understood that… and we still do not have it in FFI. But with Igor we talked the other day about it, and he needed to implement it in NativeBoost to speed up display rendering (when replacing VM display with SDL display).
So I will need to implement it as an FFI extension too… but I’m still not there (close, but not there).
What I implemented for now is just a way to avoid structure copying when accessing nested structures… something that was making the access style “a.b.c = 42” fail :)
Esteban
>
>
>
>>
>> Nicolas
>>>
>>> cheers!
>>> Esteban
>>>
>>>
>>>>
>>>> Esteban
>>>>
>>>>>
>>>>> Cheers,
>>>>> Max
>>>>>
>>>>>>
>>>>>> cheers,
>>>>>> Esteban
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20150821/ea557e92/attachment-0001.htm
More information about the Vm-dev
mailing list