> Btw: Has anyone experimented with a code generator that can transform 
> dynamic Smalltalk code into code that has some variation points (or 
> arguments) set to a fixed value? I am constantly finding myself in 
> writing such code generators manually after figuring out the algorithm 
> dynamically... at a not-so-good performance. =D For Squeak FFI, we 
> have several places where variations occur between platforms but not 
> while running on a single platform. The dynamic form looks nice but is 
> slow. The static form is tricky to maintain but fast. Hmm... one of 
> the next challenges is to make FFICallbacks fast again by generating 
> callback handlers specific to argument types and platform. You can 
> already write those by hand but this should not be necessary.
Hmm, can you give an example because I don't get it :D.

For writing generators (and transforming things in general): how about 
OMeta2? It has been long forgotten by Squeak community (and its author) 
but I think we should dust it off and give it a new life (possibly beef 
it up a little :D - add syntax highlighting etc.).

Unfortunalely the last version refused to work on Squeak 5.3 or 6.0 
(I've posted about this issue several months ago). I managed to make it 
work... kind of. I made a rudimentary Monticello package. I've attached 
it to my message along with Postload one (not mine).

Instructions how to load:

 1. Load OMeta2-Preload-MO.16.
 2. Load OMeta2-Postload-hmm.12 (You can also find this one on
    SqueakSource OMeta page. I attached it for convenience. I hope I
    don't infringe some copyrights here :D).
 3. All done.
 4. If it complains about missing #initInput method then load Postload,
    then load Preload again.

Tested on Squeak 6.0 21594 (Win 10).

I'm not sure how to add syntax highlighting to it or at least disable it 
for OMeta2Base-derived classes. That red-colored text in rule-like 
methods (because it's not valid ST code) is getting annoying.

Michał

W dniu 22.04.2022 o 14:17, Marcel Taeumel pisze:
> > But do know that at least Marcel and myself are interested in 
> enhancing the FFI significantly.
>
> Yes. I will make sure to tag a stable FFI version for Squeak 6.0. It 
> looks like we have a tagged stable FFI-Plugin and OSVM already.
>
> After the release, we will be more experimental again in both 
> SqueakFFI and ThreadedFFIPlugin. :-)
>
> Btw: Has anyone experimented with a code generator that can transform 
> dynamic Smalltalk code into code that has some variation points (or 
> arguments) set to a fixed value? I am constantly finding myself in 
> writing such code generators manually after figuring out the algorithm 
> dynamically... at a not-so-good performance. =D For Squeak FFI, we 
> have several places where variations occur between platforms but not 
> while running on a single platform. The dynamic form looks nice but is 
> slow. The static form is tricky to maintain but fast. Hmm... one of 
> the next challenges is to make FFICallbacks fast again by generating 
> callback handlers specific to argument types and platform. You can 
> already write those by hand but this should not be necessary.
>
> Best,
> Marcel
>>
>> Am 21.04.2022 20:19:23 schrieb Eliot Miranda <eliot.miranda at gmail.com>:
>>
>>
>>
>> On Mon, Apr 18, 2022 at 3:05 AM Marcel Taeumel 
>> <marcel.taeumel at hpi.de> wrote:
>>
>>     Hi Michał --
>>
>>     > Or should I not worry about that it will perform full GC (thus
>>     Image freeze) more frequently?
>>
>>     If you pin too many objects, you will annoy the compactor. This
>>     cannot be "fixed" in a full GC as far as I know. Be sure to unpin
>>     those objects in time. Only pin objects if you have to.
>>
>>
>> That said, the memory manager does try to pin objects in a single 
>> segment, so it shouldn't get too annoyed :-).
>>
>> However, ideally the marshalling logic in the FFI plugin would do the 
>> pinning and unpinning for you, automatically. The issues here are 
>> specification and backwards compatibility.  The current marshalling 
>> semantics derive from the old "V3" VM's memory manager which didn't 
>> support pinning.  Andreas (its author) explicitly forbade passing an 
>> external pointer to an object on the heap.  With pinning support I 
>> think we would like to relax this restriction, but how far? For 
>> example, it could be that one is only allowed to pass a pointer into 
>> the heap if that pointer is to an already pinned object, which might 
>> provide more safety than allowing passing a pointer to any byte-like 
>> object.  There is also an important aliasing issue; C strings need to 
>> be null-terminated, and the size of the null marker depends on string 
>> encoding.  Currently the FFI plugin copies string parameters, assumes 
>> they're 8 bit, and null terminates the copies.  This is only 
>> appropriate in some circumstances.  In others we'd like to include 
>> the null marker in the actual object itself, pin and pass a pointer 
>> to that object. But how do we specify these strategies?
>>
>> One approach is to extend the current plugin, and experience shows 
>> that this is likely the only way to make measurable progress.  
>> Another way is to move the marshalling boundary so that more can be 
>> done at the image level, simplifying the plugin and moving these 
>> policy issues up to the image level where we can implement these 
>> various policies more effectively and flexibly.  But experience shows 
>> that aiming at the ambitious "right solution" is simply a way of 
>> grinding to a complete halt (given that we have so little resource).
>>
>> But do know that at least Marcel and myself are interested in 
>> enhancing the FFI significantly.
>>
>>
>>     Best,
>>     Marcel
>>>
>>>     Am 17.04.2022 14:07:22 schrieb Michał Olszewski
>>>     <m.olszewski at nexat.pl>:
>>>
>>>     Hi,
>>>
>>>     Thanks for the answers. I'd like to ask one more question, out
>>>     of curiosity.
>>>
>>>     Does allocating (and pinning them) a large (possibly very large)
>>>     instances of RawBitsArray (Float64, ByteArray etc. for storing,
>>>     say, textures) puts significant pressure on GC during garbage
>>>     collection? Or should I not worry about that it will perform
>>>     full GC (thus Image freeze) more frequently?
>>>
>>>     Michał
>>>
>>>     W dniu 06.04.2022 o 08:31, Marcel Taeumel pisze:
>>>>     Hi Michał --
>>>>
>>>>     > Have you considered simplifying FFI interface, where callout
>>>>     (the primitive) would only handle raw bytes (along with
>>>>     argument sizes), without type checking?
>>>>
>>>>     An approach without type checking would be the Alien FFI. Here
>>>>     is Alien: http://www.squeaksource.com/Alien
>>>>     <http://www.squeaksource.com/Alien>The required IA32ABI plugin
>>>>     is also part of the OpenSmalltalk VM.
>>>>
>>>>     Squeak FFI wants to make FFI programming more robust and better
>>>>     integrated into the object-oriented world with its tools.
>>>>     Besides these particular inconveniences you just experienced,
>>>>     the goal is to make that FFI interface with ExternalType very
>>>>     simple and straightforward. And robust. :-)
>>>>
>>>>     > Maybe some support for auto-pinning objects that are passed
>>>>     to FFI?
>>>>
>>>>     Maybe there already is around FFI callbacks. Not sure.
>>>>
>>>>     > It's not like sending #pin is a big deal.
>>>>
>>>>     It sure is. It takes time. If you, for example, want to program
>>>>     a graphics back-end through FFI you need all the performance
>>>>     you can get. And too much pinning might annoy the GC compaction
>>>>     phase. :-)
>>>>
>>>>     Best,
>>>>     Marcel
>>>>>
>>>>>     Am 05.04.2022 18:50:59 schrieb Michał Olszewski
>>>>>     <m.olszewski at nexat.pl> <mailto:m.olszewski at nexat.pl>:
>>>>>
>>>>>     Hi,
>>>>>
>>>>>>     The type model for pointer types with arity > 1 is still
>>>>>>     rudimentary. You can always go the unsafe way to code against
>>>>>>     'void*'. In your case, the type 'char*[]' gives you an array
>>>>>>     type with a 'char*' content type. Not sure whether type
>>>>>>     coercing will work during the FFI call. You can also use the
>>>>>>     'string[]' type for that matter. Again, if you get a type
>>>>>>     check error on the call, you might have to resort to 'void*'
>>>>>>     instead and pass on that ExternalAddress your have at hand.
>>>>>     Yes, I did it, sorta. I allocated external bytes, copied the
>>>>>     string to them (along with null terminator), put the handle's
>>>>>     bytes into intptr_t type (a := ExternalType intptr_t allocate.
>>>>>     a at: 1 put: h getHandle asInteger), then passed that one to
>>>>>     the call.
>>>>>
>>>>>     But I thought I could get away with only using
>>>>>     in-squeak-memory objects (i.e. no external memory allocations
>>>>>     since it's a bit of a pain to handle them across the methods).
>>>>>
>>>>>
>>>>>     I have a suggestion, it may not matter much, after all you're
>>>>>     much more familiar with Squeak internals (and far more
>>>>>     experienced in programming overall).
>>>>>
>>>>>     Have you considered simplifying FFI interface, where callout
>>>>>     (the primitive) would only handle raw bytes (along with
>>>>>     argument sizes), without type checking? Type checking &
>>>>>     marshalling would then be moved to the Image side, along with
>>>>>     some "extras" for example retrieving memory address of a
>>>>>     RawBitsArray instance. This could, for example, ease
>>>>>     implementing n-dim pointers, since all the neccessary info is
>>>>>     easily accessible & no more type juggling.
>>>>>
>>>>>     Again, just some random stupid thought, please don't take it
>>>>>     seriously :D.
>>>>>
>>>>>>     You have to do it manually if you really need to pass on
>>>>>>     object memory to a call and that callee holds on to the
>>>>>>     address after the return. No need to pin objects that are
>>>>>>     just read during the call. That is, for example, fill your
>>>>>>     RawBitsArray and pass it into the function as is. I think.
>>>>>>     Just check via #isPinned. Watch out for segfaults after GC.
>>>>>>     Might indicate that you do have to pin that object. :-D
>>>>>     OK, I pin them all anyway, just in case:). Maybe some support
>>>>>     for auto-pinning objects that are passed to FFI? It's not like
>>>>>     sending #pin is a big deal. Just really easy to forget to
>>>>>     insert 10th time the same method call (and this time it would
>>>>>     actually be the one that is mandatory...).
>>>>>
>>>>>     Michał
>>>>>
>>>>>     W dniu 2022-04-05 o 11:54, Marcel Taeumel pisze:
>>>>>>     Hi Michał --
>>>>>>
>>>>>>     > As in the title, what's the way to pass single string to External
>>>>>>     > Function that expects array of strings (char**)?
>>>>>>
>>>>>>     The type model for pointer types with arity > 1 is still
>>>>>>     rudimentary. You can always go the unsafe way to code against
>>>>>>     'void*'. In your case, the type 'char*[]' gives you an array
>>>>>>     type with a 'char*' content type. Not sure whether type
>>>>>>     coercing will work during the FFI call. You can also use the
>>>>>>     'string[]' type for that matter. Again, if you get a type
>>>>>>     check error on the call, you might have to resort to 'void*'
>>>>>>     instead and pass on that ExternalAddress your have at hand.
>>>>>>
>>>>>>     (ExternalType typeNamed: 'char*[]') explore.
>>>>>>     (ExternalType typeNamed: 'string[]') explore.
>>>>>>
>>>>>>     > I assume the new FFI interface will be released along with
>>>>>>     Squeak 6.0? :)
>>>>>>
>>>>>>     Well, I will make a tag so that everybody knows what to load
>>>>>>     in Squeak 6.0. Yet, active development will continue after
>>>>>>     that release against Trunk (e.g. 6.1alpha).
>>>>>>
>>>>>>     > are objects passed to FFI (the ones allocated
>>>>>>     > on the Squeak heap) auto-pinned or do I need pin them manually?
>>>>>>
>>>>>>     You have to do it manually if you really need to pass on
>>>>>>     object memory to a call and that callee holds on to the
>>>>>>     address after the return. No need to pin objects that are
>>>>>>     just read during the call. That is, for example, fill your
>>>>>>     RawBitsArray and pass it into the function as is. I think.
>>>>>>     Just check via #isPinned. Watch out for segfaults after GC.
>>>>>>     Might indicate that you do have to pin that object. :-D
>>>>>>
>>>>>>     Best,
>>>>>>     Marcel
>>>>>>>
>>>>>>>     Am 02.04.2022 03:34:38 schrieb Michał Olszewski
>>>>>>>     <m.olszewski at nexat.pl> <mailto:m.olszewski at nexat.pl>:
>>>>>>>
>>>>>>>     Hello,
>>>>>>>
>>>>>>>     As in the title, what's the way to pass single string to
>>>>>>>     External
>>>>>>>     Function that expects array of strings (char**)? I know that
>>>>>>>     the way to
>>>>>>>     do that in plain C would be something like this: &myString
>>>>>>>     but I have
>>>>>>>     not idea how to manipulate Squeak objects such that I can
>>>>>>>     pass their
>>>>>>>     addresses manually to FFI. I guess I have to somehow wrap
>>>>>>>     stuff into
>>>>>>>     ExternalData but how?
>>>>>>>
>>>>>>>     The image used is: Squeak 6.0 alpha 21520.
>>>>>>>
>>>>>>>     I assume the new FFI interface will be released along with
>>>>>>>     Squeak 6.0? :)
>>>>>>>
>>>>>>>     At last, quick question: are objects passed to FFI (the ones
>>>>>>>     allocated
>>>>>>>     on the Squeak heap) auto-pinned or do I need pin them manually?
>>>>>>>
>>>>>>>     Michał
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
>>
>>
>> -- 
>> _,,,^..^,,,_
>> best, Eliot
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20220423/2a03eb4f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OMeta2-Postload-hmm.12.mcz
Type: application/octet-stream
Size: 10994 bytes
Desc: not available
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20220423/2a03eb4f/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OMeta2-Preload-MO.16.mcz
Type: application/octet-stream
Size: 22586 bytes
Desc: not available
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20220423/2a03eb4f/attachment-0003.obj>