[squeak-dev] Usability of Squeak FFI for anything serious...

Phil B pbpublist at gmail.com
Thu Apr 12 19:09:54 UTC 2018


Nicolas,

I've had to deal with a variety of libraries (OpenGL, databases, etc) and
have yet to run into a situation where I can't make things work with
vanilla FFI (with one caveat: I haven't needed callback support which has
been discussed several times here and on vmdev and solutions exists, I just
haven't needed them to date.)  But this has required writing a variety of
preprocessors (for static APIs) or bridges (for dynamic) as I'm not aware
of any tooling to universally deal with headers/bridging (FFI doesn't help
you with header files etc. so you will have to deal with things like unions
in your preprocessor). 32/64-bit is pretty easily dealt with provided you
check Smalltalk>>wordSize and adjust as needed (i.e. it can be dealt with
statically or dynamically depending on your needs.)  The way to think of
FFI is as just providing the lowest level plumbing materials that allows
the image to communicate with the outside world.  (i.e. pipes, elbows,
etc.. but you have to assemble it and supply the liquid)  I'm not saying
that this is ideal... it would be great if one could point to an arbitrary
API and say 'FFI, figure it out' but thats not what exists.  What we have
currently does the job but does require some work.

Just my $0.02,
Phil

On Thu, Apr 12, 2018, 4:00 AM Nicolas Cellier <
nicolas.cellier.aka.nice at gmail.com> wrote:

>
>
> 2018-04-12 3:10 GMT+02:00 Eliot Miranda <eliot.miranda at gmail.com>:
>
>> Hi Nicolas,
>>
>> On Wed, Apr 11, 2018 at 2:23 PM, Nicolas Cellier <
>> nicolas.cellier.aka.nice at gmail.com> wrote:
>>
>>> I'm trying to see how hard it is to port my HDF5 work from VW to Squeak.
>>> It was not a pleasure in VW, because that already raised some questions
>>> despite that there is a somehow detailed DLLCC manual:
>>>
>>>
>>> https://stackoverflow.com/questions/49544642/why-cant-i-pass-an-uninterpretedbytes-to-a-void-thru-dll-c-connect
>>>
>>> https://stackoverflow.com/questions/49564024/isnt-pointer-type-checking-disabled-in-dll-c-connect-and-is-that-ok
>>>
>>
>> What are you actually trying to do?  Can you post the C definitions?  I'm
>> very familiar with both the SqueakFFI and DLLCC (as of 7.4.1) and the basic
>> differences are to do with DLLCC's better cross-platform support, in that
>> it auto-redefines typedefs on load.  But other than that I think the
>> SqueakFFI is either just as good, or in some cases better (e.g. DLLCC has a
>> bug in that it is the actual parameter that carries detailed type
>> information for a struct, whereas in Andreas' SqueakFFI it is, correctly,
>> the formal parameter).
>>
>> Hi Eliot,
> the library is huge (HDF5).
> All the interfacing code is available at cincom public store (HDF5 bundle)
> To get an idea, you can inspect a clone of HDF5 library at
> https://github.com/live-clones/hdf5/tree/develop/src
> The main structures/enum/typedefs/prototypes are in the H5*public.h files
>
> The problems are all exposed in SO questions, but I can try and priotitize.
>
> The essential barrier is that those headers use almost exclusively
> typedef'ed types.
> Library is too huge for manual subsitution; this must be mechanized.
> I would prefer some kind of support in FFI rather than having to hack own
> fragile ad-hoc pre-processor.
>
> Then I have a problem because of alignment of struct fields which is
> compact in Squeak FFI rather than conforming to platform conventions.
> Same here, manually hacking with field size is a no go, this must be
> mechanized.
>
> Then some unions are used too and there is no support at all in FFI.
>
> Then definitions like size_t are dependent on machine word size (32 or 64
> bits), and there is no support for that in FFI.
> I'll have to duplicate some work for supporting both 32 and 64 bits
> Squeak, or explore new hacks...
> But there is no support and no documentation (how to).
>
> Then while digging in source coden I did not see any double pointer
> support in struct fields.
> The example I provide on SO shows what I would qualify as a bug.
> I don't know yet whether I'll need those, but it's not good.
>
> Maybe I can shoot each problem with more or less ugly workarounds.
> At the end i can declare void * and deal with offset directly, but that's
> both:
> - a lot of work
> - a good recipe for generating very hard to find bugs - crashes if lucky,
> silent corruption if not
>  (dealing with offsets is like writing in assembler instead of C)
>
> Given the size of the library, the probability of not creating such fault
> in manual translation is 0%
> Plus, there are a few evolutions from one version of the library to the
> other...
>
> That's my definition of hard++
>
>
>>
>>>
>>>
>>> But in Squeak, it sounds like hard++:
>>>
>>>
>>> https://stackoverflow.com/questions/49782651/how-one-aligns-structure-fields-in-squeak-ffi
>>>
>>> https://stackoverflow.com/questions/49783126/how-one-defines-a-union-type-in-squeak-ffi
>>>
>>> https://stackoverflow.com/questions/49783443/how-one-defines-a-fixed-size-array-member-in-a-struct-in-squeak-ffi
>>>
>>> https://stackoverflow.com/questions/49783882/how-one-deals-with-typedefs-in-squeak-ffi
>>>
>>> https://stackoverflow.com/questions/49784253/how-one-deals-with-multiple-pointer-level-like-char-in-squeak-ffi
>>>
>>> https://stackoverflow.com/questions/49784522/how-one-supports-both-32-and-64-bits-target-in-squeak-ffi
>>>
>>> Either I'm blind, or the very light documentation that I found lacks
>>> those details
>>> http://wiki.squeak.org/squeak/2426
>>>
>>> I think that I could throw a few more questions (dealing with enums...)
>>> but I don't want to excede some unknown quota and start irritating a touchy
>>> SO moderator ;)
>>>
>>> I'm afraid that there is no answer but "you're on your own"...
>>> I'm open to studying Alien or UnifiedFFI of Pharo.
>>> Or I can try and improve FFI.
>>> At this stage, my trajectory has already been quite perturbed, and
>>> despite my knowledge of rocket science, I can see that I'm not going to put
>>> HDF5 on the intended orbit :(
>>>
>>> Thoughts?
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> _,,,^..^,,,_
>> best, Eliot
>>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20180412/c5701a97/attachment.html>


More information about the Squeak-dev mailing list