[Vm-beginners] Stability of the external plugin interface

[Philip Bernhart]

Hi Eliot,


Eliot Miranda <eliot.miranda at gmail.com> writes:

> I have just spent several months porting the 32-bit
> Qwaq/Teleplace/3DICC plug-ins to 64-bit in Terf, the
> Croquet-based business communication tool, first on macOS,
> second on Windows.  It has been an enormously expensive and slow
> process.  Part of the problem is that some plugins involve three
> languages, C in the Slang-generated wrapper, C++ in the generic Qwaq
> library and the boost support libs (queues, locking, synchronization,
> etc), and Objective-C.  Part of the problem is using low-level
> debuggers for the static languages involved.  It is incredibly slow
> and costly work.
>
> Ronie has had to implement a new rendering framework because (I think,
> Ronie can explain better than me), Apple is not interested in
> supporting OpenGL in 64-bits as the world transitions to graphics
> co-processing for rendering, and also because he can do a better job
> himself than being concerned strained by an aging API such as OpenGL.
>
> Apart from the fact that Ronie is a better programmer than I, Ronie’s
> productivity has been much higher. I was porting existing code, and he
> was writing new code.  I was working with low-level code expressed as
> plugins. He had built a meta-programming facility in Smalltalk and
> generates an FFI binding and a backend composed of gpu-specific back
> end dlls in C++.
>
> Apart from security and packaging, plugins are a disaster: - having to
> generate the plugin wrapper means two focii for development, in
> Smalltalk and on the platform (Ronie’s approach has the same issue but
> not so bad because apart from building and debugging the gpu-specific
> dlls the focus is in Smalltalk, where it should be).
>
>- architecturally plug-ins force inefficiencies due to marshaling across
> the Smalltalk/platform boundary and force data copying (we have
> pinning now, so copying can and should be reduced, in using plugins or
> using the FFI, but the Qwaq plugins were written before pinning was
> available).  - browsing the plugins cannot be done in a single IDE;
> there aren’t good IDEs on linux, MacOS and Windows don’t share a
> common IDE, so changing a plug-in API takes a lot of browsing effort

> - architecturally the plugin forces the definition of *three* APIs,
> the plugin API between Smalltalk and the plugin’s internals, and the
> common internal API between the marshaled Smalltalk objects in the
> Slang-generated C plugin wrapper and the platform-specific code, and
> finally between this common internal API and each platform’s
> facilities. In an FFI-based implementation there is at most one API
> per platform, and it is easier to use facilities such as pinning to
> ease passing Smallta lk objects to external code.

Ok, I think I haven't yet understood the way of how external plugins
interact with the VM. I thought I saw only one. The interpreterProxy
defined in sqVirtualMachine.h and the initialiseModule, setInterpreter,
getModuleName and shutdownModule callbacks.

> Going forward, we will have a much more effective optimizer in
> Sista/Scorch and a much more flexible threaded FFI, along with an
> incremental global garbage collector.  This should provide facilities
> that make it feasible to move more computation into Smalltalk. Within
> Smalltalk the facilities for creating platform-neutral APIs that
> internally map to platform-specific APIs accesses through the FFI are
> much better than the static language facilities provided by
> C/C++/Objective-C et al, and the programming experience is much easier
> because there is only one foreign API to deal with, not three.  The
> rate of development is much much faster.

I'm looking forward to it, but in the meantime I need to write packages
for Cuis Smalltalk which I need which in turn go over the slow legacy
Squeak FFI with no callbacks. Is there a roadmap, some public plan of
these implementations? I think that ideas are already around a long time
already and it seems that the VM development is heavily underfunded or
else it would already be here.

> For these reasons I think it would be a huge mistake to develop any new
> rendering framework or platform interface using plugins.  The
> development should be in Smalltalk across the FFI.  Once a facility
> has been developed then one could auto generate a sandboxed
> implementation which replaces an open and potentially insecure FFI
> interface with a black box plugin, but this would be a packaging
> option, and not the main development line.

So.. if people would generate the whole "plugin" from something like
Slang and could just decide by a flag or something if the interface to
it should be over the FFI or an external plugin, would that be ok?
I have seen tries like this like the lowtalk implementation
and my current archeological project https://github.com/Phantasus/Cuis-Smalltalk-Slang/


May you feel well and safe,
Philip