[Vm-dev] nested structures with FFI?

[Esteban Lorenzano]

> On 21 Aug 2015, at 11:45, Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com> wrote:
> 
> 
> 
> 2015-08-21 5:57 GMT+02:00 Esteban Lorenzano <estebanlm at gmail.com <mailto:estebanlm at gmail.com>>:
>  
> 
>> On 21 Aug 2015, at 04:02, Esteban Lorenzano <estebanlm at gmail.com <mailto:estebanlm at gmail.com>> wrote:
>> 
>>> 
>>> On 20 Aug 2015, at 22:47, Esteban Lorenzano <estebanlm at gmail.com <mailto:estebanlm at gmail.com>> wrote:
>>> 
>>>> 
>>>> On 20 Aug 2015, at 21:48, Max Leske <maxleske at gmail.com <mailto:maxleske at gmail.com>> wrote:
>>>> 
>>>> 
>>>>> On 20 Aug 2015, at 21:28, vm-dev-request at lists.squeakfoundation.org <mailto:vm-dev-request at lists.squeakfoundation.org> wrote:
>>>>> 
>>>>> Message: 1
>>>>> Date: Thu, 20 Aug 2015 17:51:43 +0200
>>>>> From: Esteban Lorenzano <estebanlm at gmail.com <mailto:estebanlm at gmail.com>>
>>>>> Subject: [Vm-dev] nested structures with FFI?
>>>>> To: Squeak Virtual Machine Development Discussion
>>>>> 	<vm-dev at lists.squeakfoundation.org <mailto:vm-dev at lists.squeakfoundation.org>>
>>>>> Message-ID: <4F31ED4B-6246-43D6-B85B-7215D8570D28 at gmail.com <mailto:4F31ED4B-6246-43D6-B85B-7215D8570D28 at gmail.com>>
>>>>> Content-Type: text/plain; charset=utf-8
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I’m doing some tests with nested structures (to handle the conversion of NativeBoost-FFI to regular FFI) and I’m having some problems when accessing fields in the nested structure. 
>>>>> For example, this tests: 
>>>>> 
>>>>> FFITestNestingStructure class>>#fieldsDesc 
>>>>> 	^ #(
>>>>> 		byte one;
>>>>> 		FFITestStructure nested;
>>>>> 		)
>>>>> 
>>>>> FFITestStructure class>>#fieldsDesc
>>>>> 	^ #(
>>>>> 		byte byte;
>>>>> 		short short;
>>>>> 		long long;
>>>>> 		float float;
>>>>> 		double double;
>>>>> 		int64 int64;
>>>>> 		)
>>>>> 
>>>>> (definition is NB-compatible, so do not take the differences with regular FFI into account)
>>>>> 
>>>>> testNestedStructure
>>>>> 	| s1 |
>>>>> 	
>>>>> 	s1 := FFITestNestingStructure new.
>>>>> 	s1 nested byte: 42.	
>>>>> 	self assert: s1 nested byte = 42
>>>>> 
>>>>> testExternallyAllocatedNestedStructure
>>>>> 	| s1 |
>>>>> 	
>>>>> 	s1 := FFITestNestingStructure externalNew.
>>>>> 	self assert: s1 getHandle class = ExternalAddress.
>>>>> 	s1 nested byte: 42.	
>>>>> 	self assert: s1 nested byte = 42
>>>>> 
>>>>> 
>>>>> fails always because 
>>>>> 
>>>>> s1 nested byte = 0. 
>>>>> 
>>>>> (which is obviously bad). 
>>>>> 
>>>>> I remember Eliot saying that structures with FFI was not being optimal, and I wonder (before start digging more in deep) if this could have something to do?
>>>> 
>>>> I use nested structs in NB for the libgit2 bindings and they work as advertised. There can be problems if some structure is packed (i.e. uses non-4-byte alignment for some field) but that shouldn’t apply in your test case.
>>>> 
>>>> Not sure if this helps you in any way but I thought I’d let you know :)
>>> 
>>> I do not think so :)
>>> I’m quite sure this is not going to be a problem when using the ThreadedFFI backend instead ASMJIT… now my problem is with this backend, for the moment :) 
>> 
>> I’m just realised that this behaviour is correct according current ExternalStructure implementation, because:
>> 
>> nested
>> 	"This method was automatically generated"
>> 	^FFITestStructure fromHandle: (handle structAt: 2 length: 27)
>> 
>> and 
>> 
>> ByteArray>>#structAt: byteOffset length: length
>> 	"Return a structure of the given length starting at the indicated byte offset."
>> 	| value |
>> 	value := ByteArray new: length.
>> 	1 to: length do:[:i|
>> 		value unsignedByteAt: i put: (self unsignedByteAt: byteOffset+i-1)].
>> 	^value
>> 
>> So it is giving me always a copy… 
>> So we need a better solution (a non copy way to access)? I’m thinking on passing a “MappedByteArray” something that can work with the same original ByteArray (or ExternalAddress, is the same), to affect the referenced data… what do you think?
> 
> So yes, introducing a "referenced handle” (who applies offsets to same memory address or byte array) makes everything work… and looks like the correct way to handle this. 
> Do you want I commit this changes to FFI package or I keep them in my own stuff?
> 
> Esteban
> 
> 
> Yes!
> 
> years ago when I ported Smallapack to Squeak, that just was the missing sugar that would have enabled implementing all using Smalltalk memory...
> 
> Instead of that, I had to copy data to external heap in order to perform certain operations.
> For example, for extracting the imaginary part of a complex matrix interlaced representation { re[0] , im[0] , re[1] , im[1], ... } one trick is to cast a DoubleComplexMatrix * into a DoubleMatrix *, and start the copy at doubleMatrixPointer+1, with increment=2…

yes, but you are talking of something more complex than what I did, I think (what I did just works for nested structures)… what you want is a functionality I also believe I will need to add anyway :)
so I will keep playing with this and will commit when ready.

cheers!
Esteban

> 
> Nicolas
>> 
>> cheers!
>> Esteban
>>  
>> 
>>> 
>>> Esteban
>>> 
>>>> 
>>>> Cheers,
>>>> Max
>>>> 
>>>>> 
>>>>> cheers, 
>>>>> Esteban

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/vm-dev/attachments/20150821/9621d3b7/attachment-0001.htm