[squeak-dev] FFI question: array of structures as parameter

Nicolas Cellier nicolas.cellier.aka.nice at gmail.com
Tue Jun 12 20:42:31 UTC 2012


2012/6/12 Douglas McPherson <djm1329 at san.rr.com>:
> Awesome. I took at a look at ExternalArray and its subclasses. This is
> exactly what I was looking for! In fact I think the implementation of
> ExternalDoubleComplexArray does the exact type of thing I need, which is
> provide the ability to marshal an array of ExternalStructure to/from an
> underlying byte array.
>
> I've never looked at Smallapack till now. Do you think that ExternalArray
> etc should be made part of FFI, since it has uses beyond just Smallapack?
> Are there caveats? I notice that many methods have a comment like "Use with
> caution ! Might result in segmentation fault". Is this because of
> verification limitations, alignment issues, or are there other concerns as
> well?
>

Part of FFI? I never thought of it, but why not...

Caveats? obviously, passing an unbounded pointer as parameter to an
external function is not going to be fool proof any time soon...
VM isn't going to enforce anything, it's all under programmer responsibility...
Table overflow generally hang the VM quickly when happening in
VM-managed memory, but there is no guaranty, you can as well save a
corrupted image.
Table overflow in external heap might get un-noticed a bit longer, so
testing and debugging is not that easy.
Another source of error is miss-definition of external function
interface (I'm not aware of any signature verification, except maybe
the @20 suffix decoration of M$ stdcall...).
For the case of Lapack, there are many functions with many parameters,
some defining the array bounds straightly, other more indirectly...
This kind of library is exposed to both copy/paste errors and
combinatorial complexity errors (like when a function works with a
matrix transposed or not, with different bounds interpretation when it
is transposed, or with different bounds expectations for optional
outputs depending on the requested options, different work array
bounds expectations etc...). Due to work array bounds requirement
complexity, it's even not excluded that lapack parameter checking has
flaws (I think there were some corrections related to this in lapack
3.3, you can dig in http://www.netlib.org/lapack/release_notes.html).
I tried my best, and also generated automatically some low level FFi
calls, but coverage is not very high so far...

However, using bounded-garbage-collectable-memory - ByteArray would
make an ExternalArray quite safe for in-image-handling.
Except two problems...
1) The main problems I had with Smallapack was because ByteArray are 4
bytes-aligned, while some external function expect 8-byte alignment
for array of double... (I think this is related to usage of SS2/SSE3,
but not totally sure).
2) The second problem is that I can't pass the address of a sub-array with FFI
For example, {double *data=...; memmove(data+2,data,5*8);} is a FFI
call you can do with external heap pointer, but not with a ByteArray.
IMHO, this would deserve an extension to FFI, because emulating low
level operations with bounded memory is a nice property. I used this
kind of trick to handle real or imaginary part of a complex matrix
(with a strand of 2), but this also apply to other sub-matrices (a
row, a column, a sub/super diagonal).

Anyway, all code is MIT, so you're quite free to experiment and share, or not.

Nicolas

> Nice, nice :)
>
> Thanks,
> Doug
>
>
>
> On Jun 10, 2012, at 13:00 , Nicolas Cellier wrote:
>
> AFAIK, you'll have to use void * in API because arrays of structures
>
> are not supported.
>
> I mean FFI is not able to perform any type verification on array of
>
> structures...
>
>
> I created this class in Smallapack:
>
>
> ArrayedCollection subclass: #ExternalArray
>
> instanceVariableNames: 'handle size data'
>
> classVariableNames: 'EndianCache'
>
> poolDictionaries: ''
>
> category: 'Smallapack-Collection'
>
>
> - handle holds the pointer (ExternalAddress),
>
> - size the number of elements,
>
> - data is the object to be passed to the foreign function call (it
>
> holds address + type information).
>
> And here is how I had to initialize data:
>
>
> resetExternalData
>
> "Reset the ExternalData object used as FFI argument.
>
> This is cached in an inst-var instead of being recreated at each call.
>
> However, this must be reset whenever handle is changed"
>
> | cType |
>
> cType := self class type.
>
> data := cType isStructureType
>
> ifTrue:
>
> ["Arrays of structure cannot carry type checking in FFI"
>
> ExternalData fromHandle: handle type: ExternalType void]
>
> ifFalse:
>
> ["Atomic types"
>
> ExternalData fromHandle: handle type: self class type]
>
>
> Nicolas
>
>
> 2012/6/10 Douglas McPherson <djm1329 at san.rr.com>:
>
> All,
>
>
> I'm trying to call a function in an external library which takes an array of
>
> structures as one of its parameters. I can see how to define a subclass of
>
> ExternalStructure, and how that creates convenient field accessors which
>
> read/write an underlying ByteArray. I can see how to use such an
>
> ExternalStructure to pass the struct, or a pointer to it, to a function in
>
> the library. Is there a similar object I can create which allows me to
>
> read/write an array of these structs (i.e. does the proper marshaling
>
> to/from a ByteArray)?
>
>
> I got the i/f working by rolling my own -- directly filling up a ByteArray
>
> with the right number of instances of the serialized-by-hand structure,
>
> creating an ExternalData instance with said ByteArray as the handle, and
>
> passing this as the parameter. But this doesn't allow me to use the field
>
> accessor methods created by ExternalStructure, so it feels like I'm missing
>
> something.
>
>
> For concreteness, the API looks something like this:
>
>
> typedef struct
>
> {
>
> uint32_t someId;
>
> int32_t  someInt;
>
> } fooStruct;
>
>
> int32_t apiFunction
>
> (
>
> const  fooStruct *foo_list,
>
> uint32_t foo_list_size
>
> );
>
>
> Any suggestions?
>
>
> Thanks,
>
> Doug
>
>
>
>
>
>
>
>
>


More information about the Squeak-dev mailing list