On Mon, Apr 18, 2022 at 3:05 AM Marcel Taeumel <marcel.taeumel at hpi.de>
wrote:

> Hi Michał --
>
> > Or should I not worry about that it will perform full GC (thus Image
> freeze) more frequently?
>
> If you pin too many objects, you will annoy the compactor. This cannot be
> "fixed" in a full GC as far as I know. Be sure to unpin those objects in
> time. Only pin objects if you have to.
>

That said, the memory manager does try to pin objects in a single segment,
so it shouldn't get too annoyed :-).

However, ideally the marshalling logic in the FFI plugin would do the
pinning and unpinning for you, automatically.  The issues here are
specification and backwards compatibility.  The current marshalling
semantics derive from the old "V3" VM's memory manager which didn't support
pinning.  Andreas (its author) explicitly forbade passing an external
pointer to an object on the heap.  With pinning support I think we would
like to relax this restriction, but how far?  For example, it could be that
one is only allowed to pass a pointer into the heap if that pointer is to
an already pinned object, which might provide more safety than allowing
passing a pointer to any byte-like object.  There is also an important
aliasing issue; C strings need to be null-terminated, and the size of the
null marker depends on string encoding.  Currently the FFI plugin copies
string parameters, assumes they're 8 bit, and null terminates the copies.
This is only appropriate in some circumstances.  In others we'd like to
include the null marker in the actual object itself, pin and pass a pointer
to that object.  But how do we specify these strategies?

One approach is to extend the current plugin, and experience shows that
this is likely the only way to make measurable progress.  Another way is to
move the marshalling boundary so that more can be done at the image level,
simplifying the plugin and moving these policy issues up to the image level
where we can implement these various policies more effectively and
flexibly.  But experience shows that aiming at the ambitious "right
solution" is simply a way of grinding to a complete halt (given that we
have so little resource).

But do know that at least Marcel and myself are interested in enhancing the
FFI significantly.


> Best,
> Marcel
>
> Am 17.04.2022 14:07:22 schrieb Michał Olszewski <m.olszewski at nexat.pl>:
>
> Hi,
>
> Thanks for the answers. I'd like to ask one more question, out of
> curiosity.
>
> Does allocating (and pinning them) a large (possibly very large) instances
> of RawBitsArray (Float64, ByteArray etc. for storing, say, textures) puts
> significant pressure on GC during garbage collection? Or should I not worry
> about that it will perform full GC (thus Image freeze) more frequently?
>
> Michał
> W dniu 06.04.2022 o 08:31, Marcel Taeumel pisze:
>
> Hi Michał --
>
> > Have you considered simplifying FFI interface, where callout (the
> primitive) would only handle raw bytes (along with argument sizes), without
> type checking?
>
> An approach without type checking would be the Alien FFI. Here is Alien:
> http://www.squeaksource.com/Alien The required IA32ABI plugin is also
> part of the OpenSmalltalk VM.
>
> Squeak FFI wants to make FFI programming more robust and better integrated
> into the object-oriented world with its tools. Besides these particular
> inconveniences you just experienced, the goal is to make that FFI interface
> with ExternalType very simple and straightforward. And robust. :-)
>
> > Maybe some support for auto-pinning objects that are passed to FFI?
>
> Maybe there already is around FFI callbacks. Not sure.
>
> > It's not like sending #pin is a big deal.
>
> It sure is. It takes time. If you, for example, want to program a graphics
> back-end through FFI you need all the performance you can get. And too much
> pinning might annoy the GC compaction phase. :-)
>
> Best,
> Marcel
>
> Am 05.04.2022 18:50:59 schrieb Michał Olszewski <m.olszewski at nexat.pl>
> <m.olszewski at nexat.pl>:
>
> Hi,
>
> The type model for pointer types with arity > 1 is still rudimentary. You
> can always go the unsafe way to code against 'void*'. In your case, the
> type 'char*[]' gives you an array type with a 'char*' content type. Not
> sure whether type coercing will work during the FFI call. You can also use
> the 'string[]' type for that matter. Again, if you get a type check error
> on the call, you might have to resort to 'void*' instead and pass on that
> ExternalAddress your have at hand.
>
> Yes, I did it, sorta. I allocated external bytes, copied the string to
> them (along with null terminator), put the handle's bytes into intptr_t
> type (a := ExternalType intptr_t allocate. a at: 1 put: h getHandle
> asInteger), then passed that one to the call.
>
> But I thought I could get away with only using in-squeak-memory objects
> (i.e. no external memory allocations since it's a bit of a pain to handle
> them across the methods).
>
>
> I have a suggestion, it may not matter much, after all you're much more
> familiar with Squeak internals (and far more experienced in programming
> overall).
>
> Have you considered simplifying FFI interface, where callout (the
> primitive) would only handle raw bytes (along with argument sizes), without
> type checking? Type checking & marshalling would then be moved to the Image
> side, along with some "extras" for example retrieving memory address of a
> RawBitsArray instance. This could, for example, ease implementing n-dim
> pointers, since all the neccessary info is easily accessible & no more type
> juggling.
>
> Again, just some random stupid thought, please don't take it seriously :D.
>
> You have to do it manually if you really need to pass on object memory to
> a call and that callee holds on to the address after the return. No need to
> pin objects that are just read during the call. That is, for example, fill
> your RawBitsArray and pass it into the function as is. I think. Just check
> via #isPinned. Watch out for segfaults after GC. Might indicate that you do
> have to pin that object. :-D
>
> OK, I pin them all anyway, just in case:). Maybe some support for
> auto-pinning objects that are passed to FFI? It's not like sending #pin is
> a big deal. Just really easy to forget to insert 10th time the same method
> call (and this time it would actually be the one that is mandatory...).
>
> Michał
> W dniu 2022-04-05 o 11:54, Marcel Taeumel pisze:
>
> Hi Michał --
>
> > As in the title, what's the way to pass single string to External
> > Function that expects array of strings (char**)?
>
> The type model for pointer types with arity > 1 is still rudimentary. You
> can always go the unsafe way to code against 'void*'. In your case, the
> type 'char*[]' gives you an array type with a 'char*' content type. Not
> sure whether type coercing will work during the FFI call. You can also use
> the 'string[]' type for that matter. Again, if you get a type check error
> on the call, you might have to resort to 'void*' instead and pass on that
> ExternalAddress your have at hand.
>
> (ExternalType typeNamed: 'char*[]') explore.
> (ExternalType typeNamed: 'string[]') explore.
>
> > I assume the new FFI interface will be released along with Squeak 6.0?
> :)
>
> Well, I will make a tag so that everybody knows what to load in Squeak
> 6.0. Yet, active development will continue after that release against Trunk
> (e.g. 6.1alpha).
>
> > are objects passed to FFI (the ones allocated
> > on the Squeak heap) auto-pinned or do I need pin them manually?
>
> You have to do it manually if you really need to pass on object memory to
> a call and that callee holds on to the address after the return. No need to
> pin objects that are just read during the call. That is, for example, fill
> your RawBitsArray and pass it into the function as is. I think. Just check
> via #isPinned. Watch out for segfaults after GC. Might indicate that you do
> have to pin that object. :-D
>
> Best,
> Marcel
>
> Am 02.04.2022 03:34:38 schrieb Michał Olszewski <m.olszewski at nexat.pl>
> <m.olszewski at nexat.pl>:
> Hello,
>
> As in the title, what's the way to pass single string to External
> Function that expects array of strings (char**)? I know that the way to
> do that in plain C would be something like this: &myString but I have
> not idea how to manipulate Squeak objects such that I can pass their
> addresses manually to FFI. I guess I have to somehow wrap stuff into
> ExternalData but how?
>
> The image used is: Squeak 6.0 alpha 21520.
>
> I assume the new FFI interface will be released along with Squeak 6.0? :)
>
> At last, quick question: are objects passed to FFI (the ones allocated
> on the Squeak heap) auto-pinned or do I need pin them manually?
>
> Michał
>
>
>
>
>
>

-- 
_,,,^..^,,,_
best, Eliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20220421/92024ebb/attachment.html>