On 12 June 2012 22:42, Nicolas Cellier
<nicolas.cellier.aka.nice at gmail.com> wrote:
> 2012/6/12 Douglas McPherson <djm1329 at san.rr.com>:
>> Awesome. I took at a look at ExternalArray and its subclasses. This is
>> exactly what I was looking for! In fact I think the implementation of
>> ExternalDoubleComplexArray does the exact type of thing I need, which is
>> provide the ability to marshal an array of ExternalStructure to/from an
>> underlying byte array.
>>
>> I've never looked at Smallapack till now. Do you think that ExternalArray
>> etc should be made part of FFI, since it has uses beyond just Smallapack?
>> Are there caveats? I notice that many methods have a comment like "Use with
>> caution ! Might result in segmentation fault". Is this because of
>> verification limitations, alignment issues, or are there other concerns as
>> well?
>>
>
> Part of FFI? I never thought of it, but why not...
>
> Caveats? obviously, passing an unbounded pointer as parameter to an
> external function is not going to be fool proof any time soon...
> VM isn't going to enforce anything, it's all under programmer responsibility...
> Table overflow generally hang the VM quickly when happening in
> VM-managed memory, but there is no guaranty, you can as well save a
> corrupted image.
> Table overflow in external heap might get un-noticed a bit longer, so
> testing and debugging is not that easy.
> Another source of error is miss-definition of external function
> interface (I'm not aware of any signature verification, except maybe
> the @20 suffix decoration of M$ stdcall...).
> For the case of Lapack, there are many functions with many parameters,
> some defining the array bounds straightly, other more indirectly...
> This kind of library is exposed to both copy/paste errors and
> combinatorial complexity errors (like when a function works with a
> matrix transposed or not, with different bounds interpretation when it
> is transposed, or with different bounds expectations for optional
> outputs depending on the requested options, different work array
> bounds expectations etc...). Due to work array bounds requirement
> complexity, it's even not excluded that lapack parameter checking has
> flaws (I think there were some corrections related to this in lapack
> 3.3, you can dig in http://www.netlib.org/lapack/release_notes.html).
> I tried my best, and also generated automatically some low level FFi
> calls, but coverage is not very high so far...
>
> However, using bounded-garbage-collectable-memory - ByteArray would
> make an ExternalArray quite safe for in-image-handling.
> Except two problems...
> 1) The main problems I had with Smallapack was because ByteArray are 4
> bytes-aligned, while some external function expect 8-byte alignment
> for array of double... (I think this is related to usage of SS2/SSE3,
> but not totally sure).
> 2) The second problem is that I can't pass the address of a sub-array with FFI
> For example, {double *data=...; memmove(data+2,data,5*8);} is a FFI
> call you can do with external heap pointer, but not with a ByteArray.
> IMHO, this would deserve an extension to FFI, because emulating low
> level operations with bounded memory is a nice property. I used this
> kind of trick to handle real or imaginary part of a complex matrix
> (with a strand of 2), but this also apply to other sub-matrices (a
> row, a column, a sub/super diagonal).
>

Some advertisement:
with NativeBoost you can deal with 2) quite easily
and even with 1) (just leave some space in the bytearray for
alignment, then during callout NB can realign the data in array if
it's not aligned properly)
But that, of course will require some ass embler hacking :)

> Anyway, all code is MIT, so you're quite free to experiment and share, or not.
>
> Nicolas
>


-- 
Best regards,
Igor Stasenko.