<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr">Hi Philip, Hi All,</div><div dir="ltr"><br><blockquote type="cite">On May 24, 2020, at 2:37 PM, Philip Bernhart <philip.bernhart@posteo.de> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><span>Hi Eliot,</span><br><span></span><br><span>Eliot Miranda <eliot.miranda@gmail.com> writes:</span><br><span></span><br><blockquote type="cite"><span>Hi Philip,</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>On Sat, May 23, 2020 at 9:27 AM Philip Bernhart <philip.bernhart@posteo.de></span><br></blockquote><blockquote type="cite"><span>wrote:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><blockquote type="cite"><span>Hello,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>I want to write an external plugin - I think,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>for having a fast interface to a C library.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span></span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>How is that currently done?</span><br></blockquote></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Either by writing a plugin or directly through the FFI. We can discuss how</span><br></blockquote><blockquote type="cite"><span>to make the choice between the two but I lean towards the FFI.</span><br></blockquote><span></span><br><span>Why is that? I see the case for calling external libraries in a more</span><br><span>general fashion FFI. I see why people don't want to have additional</span><br><span>plugins in the VM too, more stuff to maintain -> more bugs, also some</span><br><span>VM internal resource exhaustion issues associated with internal</span><br><span>plugins(?), etc.</span><br><span>I'm not asking for that, I'm talking about external plugins.</span><br><span></span><br><span>I read a little bit the Swiki articles about FFI / Plugins and people</span><br><span>who wrote them generally seemed to lean toward plugins, when it comes to</span><br><span>a layer provided to the users of the functionality which is supposed</span><br><span>to be put into the plugin / library. And they seemed to be a little</span><br><span>bit faster than the FFI interface.</span><br><span></span><br><span>So what are your reasons for prefering FFI over External Plugins? :-)</span><br></div></blockquote><div><br></div>here is an extract of a recent conversation with Marcel Taumael and others that started off-line. It states my reasons for preferring FFI over plugins.<div><br></div><div><span style="-webkit-text-size-adjust: auto;">———8<———</span></div><div><div dir="ltr" style="-webkit-text-size-adjust: auto;"><blockquote type="cite">On May 13, 2020, at 5:58 AM, Marcel Taeumel <<a href="mailto:marcel.taeumel@hpi.de" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="1" style="color: currentcolor;">marcel.taeumel@hpi.de</a>> wrote:<br><br></blockquote></div><blockquote type="cite" style="color: rgb(0, 0, 0); -webkit-text-size-adjust: auto;"><div dir="ltr"><div id="__MailbirdStyleContent" style="font-size: 10pt; font-family: Arial;">Hi Nicolas, hi Eliot, hi Levente, hi Vanessa.<div class="mb_sig"></div><div><br></div><div>What is your opinion about "SDL through FFI" as it is proposed as the big thing in the "minheadless" branch and heavily used in recent Pharo versions?</div><div><br></div><div>I think that</div><div>- Yes, it is a good idea to reduce the need for platform-specific code.</div><div>- No, it is a terrible idea to let SDL (or any C) internals bleed in to the regular Squeak/Smalltalk image.</div><div>- Instead, we should design a simple "WindowPlugin" that can easily map to SDL in a cross-platform fashion. Such a plugin can be extended to provide debugging support in a domain-specific way.</div><div>- However, it should always remain possible to provide alternative backends to SDL (such as Qt or native OS libs) for experimentation or workarounds or bugfixes. Just like the "Project" abstraction in Squeak to keep MVC and Morphic and whatever around at the same time.</div><div><br></div><div>Isn't that what the unix part is doing anyway? vm-display-X11, vm-display-Quartz ...</div></div></div></blockquote><div style="-webkit-text-size-adjust: auto;"><br></div><span style="-webkit-text-size-adjust: auto; background-color: rgb(255, 255, 255);">I have just spent several months porting the 32-bit Qwaq/Teleplace/3DICC plug-ins to 64-bit in Terf, the Croquet-based business communication tool, first on macOS, second on Windows. It has been an enormously expensive and slow process. Part of the problem is that some plugins involve three languages, C in the Slang-generated wrapper, C++ in the generic Qwaq library and the boost support libs (queues, locking, synchronization, etc), and Objective-C. Part of the problem is using low-level debuggers for the static languages involved. It is incredibly slow and costly work.</span><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">Ronie has had to implement a new rendering framework because (I think, Ronie can explain better than me), Apple is not interested in supporting OpenGL in 64-bits as the world transitions to graphics co-processing for rendering, and also because he can do a better job himself than being concerned strained by an aging API such as OpenGL.</div><div style="-webkit-text-size-adjust: auto;"><br></div><div style="-webkit-text-size-adjust: auto;">Apart from the fact that Ronie is a better programmer than I, Ronie’s productivity has been much higher. I was porting existing code, and he was writing new code. I was working with low-level code expressed as plugins. He had built a meta-programming facility in Smalltalk and generates an FFI binding and a backend composed of gpu-specific back end dlls in C++.<br><div><br></div><div>Apart from security and packaging, plugins are a disaster:</div><div>- having to generate the plugin wrapper means two focii for development, in Smalltalk and on the platform (Ronie’s approach has the same issue but not so bad because apart from building and debugging the gpu-specific dlls the focus is in Smalltalk, where it should be).</div><div>- architecturally plug-ins force inefficiencies due to marshaling across the Smalltalk/platform boundary and force data copying (we have pinning now, so copying can and should be reduced, in using plugins or using the FFI, but the Qwaq plugins were written before pinning was available).</div><div>- browsing the plugins cannot be done in a single IDE; there aren’t good IDEs on linux, MacOS and Windows don’t share a common IDE, so changing a plug-in API takes a lot of browsing effort</div><div>- architecturally the plugin forces the definition of *three* APIs, the plugin API between Smalltalk and the plugin’s internals, and the common internal API between the marshaled Smalltalk objects in the Slang-generated C plugin wrapper and the platform-specific code, and finally between this common internal API and each platform’s facilities. In an FFI-based implementation there is at most one API per platform, and it is easier to use facilities such as pinning to ease passing Smalltalk objects to external code.</div><div><br></div><div>Going forward, we will have a much more effective optimizer in Sista/Scorch and a much more flexible threaded FFI, along with an incremental global garbage collector. This should provide facilities that make it feasible to move more computation into Smalltalk. Within Smalltalk the facilities for creating platform-neutral APIs that internally map to platform-specific APIs accesses through the FFI are much better than the static language facilities provided by C/C++/Objective-C et al, and the programming experience is much easier because there is only one foreign API to deal with, not three. The rate of development is much much faster.</div><div><br></div><div>If we stay with plugins we fall into the Python trap. The plugin legacy is hugely expensive and restrictive, and prevents us from evolving, both because there is substantial investment and performance in the plugins and because they are inherently more viscous (for the 3 API reasons above).</div><div><br></div><div>For these reasons I think it would be a huge mistake to develop any new rendering framework or platform interface using plugins. The development should be in Smalltalk across the FFI. Once a facility has been developed then one could auto generate a sandboxed implementation which replaces an open and potentially insecure FFI interface with a black box plugin, but this would be a packaging option, and not the main development line.</div><div><br></div><div>Further, we could perhaps develop a different style of security, building on the ideas of the SecurityPlugin. Perhaps we could parameterize the FFI so it will only allow certain functions to be called. Imagine for example an FFISecurityPlugin which is built statically that combines a list of names (the names of the foreign functions that the image is allowed to call), with the dlls the FFI is allowed to use (either the dlls themselves or their names). Attempts to use foreign functions or dlls outside the names in the predefined set, would be outlawed.</div><div><br></div><div>So my experience, not just the recent experience with 3D ICC, but also many years with VisualWorks, and some before with BrouHaHa, leads me to push strongly for embracing the FFI and addressing security issues directly, without tying our hands and putting an albatross around our necks to weigh us down.</div><div><br></div><div>Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology. The world looks different now. We should evolve to match the new environment. I agree that security is very important. But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.</div><div>———8<———</div><div><br></div></div><blockquote type="cite"><div dir="ltr"><span></span><br><span></span><br><span>Thanks for your time,</span><br><span>Philip</span><br></div></blockquote></div></body></html>