[squeak-dev] Usability of Squeak FFI for anything serious...

Nicolas Cellier nicolas.cellier.aka.nice at gmail.com
Thu Apr 12 20:34:58 UTC 2018


2018-04-12 21:09 GMT+02:00 Phil B <pbpublist at gmail.com>:

> Nicolas,
>
> I've had to deal with a variety of libraries (OpenGL, databases, etc) and
> have yet to run into a situation where I can't make things work with
> vanilla FFI (with one caveat: I haven't needed callback support which has
> been discussed several times here and on vmdev and solutions exists, I just
> haven't needed them to date.)  But this has required writing a variety of
> preprocessors (for static APIs) or bridges (for dynamic) as I'm not aware
> of any tooling to universally deal with headers/bridging (FFI doesn't help
> you with header files etc. so you will have to deal with things like unions
> in your preprocessor). 32/64-bit is pretty easily dealt with provided you
> check Smalltalk>>wordSize and adjust as needed (i.e. it can be dealt with
> statically or dynamically depending on your needs.)  The way to think of
> FFI is as just providing the lowest level plumbing materials that allows
> the image to communicate with the outside world.  (i.e. pipes, elbows,
> etc.. but you have to assemble it and supply the liquid)  I'm not saying
> that this is ideal... it would be great if one could point to an arbitrary
> API and say 'FFI, figure it out' but thats not what exists.  What we have
> currently does the job but does require some work.
>
> Just my $0.02,
> Phil
>
>
$0.02 taken :)
There are at least two problems:
- documentation (lack of)
- limitations (unions, pointer on pointer on ...)
- automation (I said at least two, not at most)

For the first problem, I opened the SO issues. Not ideal, but it's a start.
For the second, I'll see if I can do something...
For the third, I did import some .h header with VW DLLCC, and I'm in the
process of translating to other dialects.
It's easier because Smalltalk syntax is order of magnitudes smaller than C
syntax.

cheers


>
> On Thu, Apr 12, 2018, 4:00 AM Nicolas Cellier <nicolas.cellier.aka.nice@
> gmail.com> wrote:
>
>>
>>
>> 2018-04-12 3:10 GMT+02:00 Eliot Miranda <eliot.miranda at gmail.com>:
>>
>>> Hi Nicolas,
>>>
>>> On Wed, Apr 11, 2018 at 2:23 PM, Nicolas Cellier <
>>> nicolas.cellier.aka.nice at gmail.com> wrote:
>>>
>>>> I'm trying to see how hard it is to port my HDF5 work from VW to Squeak.
>>>> It was not a pleasure in VW, because that already raised some questions
>>>> despite that there is a somehow detailed DLLCC manual:
>>>>
>>>> https://stackoverflow.com/questions/49544642/why-cant-i-
>>>> pass-an-uninterpretedbytes-to-a-void-thru-dll-c-connect
>>>> https://stackoverflow.com/questions/49564024/isnt-
>>>> pointer-type-checking-disabled-in-dll-c-connect-and-is-that-ok
>>>>
>>>
>>> What are you actually trying to do?  Can you post the C definitions?
>>> I'm very familiar with both the SqueakFFI and DLLCC (as of 7.4.1) and the
>>> basic differences are to do with DLLCC's better cross-platform support, in
>>> that it auto-redefines typedefs on load.  But other than that I think the
>>> SqueakFFI is either just as good, or in some cases better (e.g. DLLCC has a
>>> bug in that it is the actual parameter that carries detailed type
>>> information for a struct, whereas in Andreas' SqueakFFI it is, correctly,
>>> the formal parameter).
>>>
>>> Hi Eliot,
>> the library is huge (HDF5).
>> All the interfacing code is available at cincom public store (HDF5 bundle)
>> To get an idea, you can inspect a clone of HDF5 library at
>> https://github.com/live-clones/hdf5/tree/develop/src
>> The main structures/enum/typedefs/prototypes are in the H5*public.h files
>>
>> The problems are all exposed in SO questions, but I can try and
>> priotitize.
>>
>> The essential barrier is that those headers use almost exclusively
>> typedef'ed types.
>> Library is too huge for manual subsitution; this must be mechanized.
>> I would prefer some kind of support in FFI rather than having to hack own
>> fragile ad-hoc pre-processor.
>>
>> Then I have a problem because of alignment of struct fields which is
>> compact in Squeak FFI rather than conforming to platform conventions.
>> Same here, manually hacking with field size is a no go, this must be
>> mechanized.
>>
>> Then some unions are used too and there is no support at all in FFI.
>>
>> Then definitions like size_t are dependent on machine word size (32 or 64
>> bits), and there is no support for that in FFI.
>> I'll have to duplicate some work for supporting both 32 and 64 bits
>> Squeak, or explore new hacks...
>> But there is no support and no documentation (how to).
>>
>> Then while digging in source coden I did not see any double pointer
>> support in struct fields.
>> The example I provide on SO shows what I would qualify as a bug.
>> I don't know yet whether I'll need those, but it's not good.
>>
>> Maybe I can shoot each problem with more or less ugly workarounds.
>> At the end i can declare void * and deal with offset directly, but that's
>> both:
>> - a lot of work
>> - a good recipe for generating very hard to find bugs - crashes if lucky,
>> silent corruption if not
>>  (dealing with offsets is like writing in assembler instead of C)
>>
>> Given the size of the library, the probability of not creating such fault
>> in manual translation is 0%
>> Plus, there are a few evolutions from one version of the library to the
>> other...
>>
>> That's my definition of hard++
>>
>>
>>>
>>>>
>>>>
>>>> But in Squeak, it sounds like hard++:
>>>>
>>>> https://stackoverflow.com/questions/49782651/how-one-
>>>> aligns-structure-fields-in-squeak-ffi
>>>> https://stackoverflow.com/questions/49783126/how-one-
>>>> defines-a-union-type-in-squeak-ffi
>>>> https://stackoverflow.com/questions/49783443/how-one-
>>>> defines-a-fixed-size-array-member-in-a-struct-in-squeak-ffi
>>>> https://stackoverflow.com/questions/49783882/how-one-
>>>> deals-with-typedefs-in-squeak-ffi
>>>> https://stackoverflow.com/questions/49784253/how-one-
>>>> deals-with-multiple-pointer-level-like-char-in-squeak-ffi
>>>> https://stackoverflow.com/questions/49784522/how-one-
>>>> supports-both-32-and-64-bits-target-in-squeak-ffi
>>>>
>>>> Either I'm blind, or the very light documentation that I found lacks
>>>> those details
>>>> http://wiki.squeak.org/squeak/2426
>>>>
>>>> I think that I could throw a few more questions (dealing with enums...)
>>>> but I don't want to excede some unknown quota and start irritating a touchy
>>>> SO moderator ;)
>>>>
>>>> I'm afraid that there is no answer but "you're on your own"...
>>>> I'm open to studying Alien or UnifiedFFI of Pharo.
>>>> Or I can try and improve FFI.
>>>> At this stage, my trajectory has already been quite perturbed, and
>>>> despite my knowledge of rocket science, I can see that I'm not going to put
>>>> HDF5 on the intended orbit :(
>>>>
>>>> Thoughts?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> _,,,^..^,,,_
>>> best, Eliot
>>>
>>>
>>>
>>>
>>
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20180412/7c47dbbd/attachment.html>


More information about the Squeak-dev mailing list