Hi Philip, Hi All,
On May 24, 2020, at 2:37 PM, Philip Bernhart philip.bernhart@posteo.de wrote:
Hi Eliot,
Eliot Miranda eliot.miranda@gmail.com writes:
Hi Philip,
On Sat, May 23, 2020 at 9:27 AM Philip Bernhart philip.bernhart@posteo.de wrote:
Hello,
I want to write an external plugin - I think, for having a fast interface to a C library.
How is that currently done?
Either by writing a plugin or directly through the FFI. We can discuss how to make the choice between the two but I lean towards the FFI.
Why is that? I see the case for calling external libraries in a more general fashion FFI. I see why people don't want to have additional plugins in the VM too, more stuff to maintain -> more bugs, also some VM internal resource exhaustion issues associated with internal plugins(?), etc. I'm not asking for that, I'm talking about external plugins.
I read a little bit the Swiki articles about FFI / Plugins and people who wrote them generally seemed to lean toward plugins, when it comes to a layer provided to the users of the functionality which is supposed to be put into the plugin / library. And they seemed to be a little bit faster than the FFI interface.
So what are your reasons for prefering FFI over External Plugins? :-)
here is an extract of a recent conversation with Marcel Taumael and others that started off-line. It states my reasons for preferring FFI over plugins.
———8<———
On May 13, 2020, at 5:58 AM, Marcel Taeumel marcel.taeumel@hpi.de wrote:
Hi Nicolas, hi Eliot, hi Levente, hi Vanessa.
What is your opinion about "SDL through FFI" as it is proposed as the big thing in the "minheadless" branch and heavily used in recent Pharo versions?
I think that
- Yes, it is a good idea to reduce the need for platform-specific code.
- No, it is a terrible idea to let SDL (or any C) internals bleed in to the regular Squeak/Smalltalk image.
- Instead, we should design a simple "WindowPlugin" that can easily map to SDL in a cross-platform fashion. Such a plugin can be extended to provide debugging support in a domain-specific way.
- However, it should always remain possible to provide alternative backends to SDL (such as Qt or native OS libs) for experimentation or workarounds or bugfixes. Just like the "Project" abstraction in Squeak to keep MVC and Morphic and whatever around at the same time.
Isn't that what the unix part is doing anyway? vm-display-X11, vm-display-Quartz ...
I have just spent several months porting the 32-bit Qwaq/Teleplace/3DICC plug-ins to 64-bit in Terf, the Croquet-based business communication tool, first on macOS, second on Windows. It has been an enormously expensive and slow process. Part of the problem is that some plugins involve three languages, C in the Slang-generated wrapper, C++ in the generic Qwaq library and the boost support libs (queues, locking, synchronization, etc), and Objective-C. Part of the problem is using low-level debuggers for the static languages involved. It is incredibly slow and costly work.
Ronie has had to implement a new rendering framework because (I think, Ronie can explain better than me), Apple is not interested in supporting OpenGL in 64-bits as the world transitions to graphics co-processing for rendering, and also because he can do a better job himself than being concerned strained by an aging API such as OpenGL.
Apart from the fact that Ronie is a better programmer than I, Ronie’s productivity has been much higher. I was porting existing code, and he was writing new code. I was working with low-level code expressed as plugins. He had built a meta-programming facility in Smalltalk and generates an FFI binding and a backend composed of gpu-specific back end dlls in C++.
Apart from security and packaging, plugins are a disaster: - having to generate the plugin wrapper means two focii for development, in Smalltalk and on the platform (Ronie’s approach has the same issue but not so bad because apart from building and debugging the gpu-specific dlls the focus is in Smalltalk, where it should be). - architecturally plug-ins force inefficiencies due to marshaling across the Smalltalk/platform boundary and force data copying (we have pinning now, so copying can and should be reduced, in using plugins or using the FFI, but the Qwaq plugins were written before pinning was available). - browsing the plugins cannot be done in a single IDE; there aren’t good IDEs on linux, MacOS and Windows don’t share a common IDE, so changing a plug-in API takes a lot of browsing effort - architecturally the plugin forces the definition of *three* APIs, the plugin API between Smalltalk and the plugin’s internals, and the common internal API between the marshaled Smalltalk objects in the Slang-generated C plugin wrapper and the platform-specific code, and finally between this common internal API and each platform’s facilities. In an FFI-based implementation there is at most one API per platform, and it is easier to use facilities such as pinning to ease passing Smalltalk objects to external code.
Going forward, we will have a much more effective optimizer in Sista/Scorch and a much more flexible threaded FFI, along with an incremental global garbage collector. This should provide facilities that make it feasible to move more computation into Smalltalk. Within Smalltalk the facilities for creating platform-neutral APIs that internally map to platform-specific APIs accesses through the FFI are much better than the static language facilities provided by C/C++/Objective-C et al, and the programming experience is much easier because there is only one foreign API to deal with, not three. The rate of development is much much faster.
If we stay with plugins we fall into the Python trap. The plugin legacy is hugely expensive and restrictive, and prevents us from evolving, both because there is substantial investment and performance in the plugins and because they are inherently more viscous (for the 3 API reasons above).
For these reasons I think it would be a huge mistake to develop any new rendering framework or platform interface using plugins. The development should be in Smalltalk across the FFI. Once a facility has been developed then one could auto generate a sandboxed implementation which replaces an open and potentially insecure FFI interface with a black box plugin, but this would be a packaging option, and not the main development line.
Further, we could perhaps develop a different style of security, building on the ideas of the SecurityPlugin. Perhaps we could parameterize the FFI so it will only allow certain functions to be called. Imagine for example an FFISecurityPlugin which is built statically that combines a list of names (the names of the foreign functions that the image is allowed to call), with the dlls the FFI is allowed to use (either the dlls themselves or their names). Attempts to use foreign functions or dlls outside the names in the predefined set, would be outlawed.
So my experience, not just the recent experience with 3D ICC, but also many years with VisualWorks, and some before with BrouHaHa, leads me to push strongly for embracing the FFI and addressing security issues directly, without tying our hands and putting an albatross around our necks to weigh us down.
Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology. The world looks different now. We should evolve to match the new environment. I agree that security is very important. But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation. ———8<———
Thanks for your time, Philip
On 2020-05-24, at 3:57 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology. The world looks different now. We should evolve to match the new environment. I agree that security is very important. But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.
The VM plugins were an attempt to get away from a few problems that plagued VM development at the end of the last millennium (and I don't mean March this year) with things like a limited number of prim table entires, the lack of flexibility a static table engendered, the time it took to compile an entire VM for a tiny change/test cycle etc (IIRC of the order of half an hour back then). The original external plugins stuff meant you could get to outside code via a name system that didn't screw the prim table (and I'd still love to find away to get rid of it entirely, but dream on) and that let you build/load/try/unload/rebuild quickly. We were also starting the beginnings of the ffi stuff around then. Andy Greenberg came up with the SmartSyntaxPlugin stuff, for example, at the end of '99. Apparently I sent Dan/Andreas a final pluginized linux vm late April 2000.
It's worth remembering just how awful Windows & Mac OS were back then as part of the wider scenario - non-preemptive processscheduling, weird callback based event reading, general nonsense. Checking ancient emails to find dates etc reminded me of a huge list of other stuff we were working on back then that *still* needs finishing! And the early pinned objects stuff... and private methods! And...!) Oh-ho - and as an amusing aside, an email from Eliot of that sort of date was pointing out how amazing it was to be able to get a 30Gb disk for a mere $190; you can get 10TB now.
I definitely agree that for an awful lot of cases using a good ffi is better nowadays. It does have some costs though; but in some case I think those are costs we should be pleased to pay. For example, a number of current plugins would be better replaced with ffi classes that actually work with the OS facilities rather than trying to work around them. The FilePlugin comes readily to mind. How much nicer it would be to handle files with direct ffi calls to OS routines and have the complicated stuff in Smalltalk bridge classes. Hmm, now where did we have this in the past; oh yes, in the BrouHaHa images 30-some years ago.
Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.
Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: DCVP: Destroy another Computer Via Phone-link
On May 24, 2020, at 8:14 PM, tim Rowledge tim@rowledge.org wrote:
On 2020-05-24, at 3:57 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology. The world looks different now. We should evolve to match the new environment. I agree that security is very important. But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.
The VM plugins were an attempt to get away from a few problems that plagued VM development at the end of the last millennium (and I don't mean March this year) with things like a limited number of prim table entires, the lack of flexibility a static table engendered, the time it took to compile an entire VM for a tiny change/test cycle etc (IIRC of the order of half an hour back then). The original external plugins stuff meant you could get to outside code via a name system that didn't screw the prim table (and I'd still love to find away to get rid of it entirely, but dream on) and that let you build/load/try/unload/rebuild quickly. We were also starting the beginnings of the ffi stuff around then. Andy Greenberg came up with the SmartSyntaxPlugin stuff, for example, at the end of '99. Apparently I sent Dan/Andreas a final pluginized linux vm late April 2000.
It's worth remembering just how awful Windows & Mac OS were back then as part of the wider scenario - non-preemptive processscheduling, weird callback based event reading, general nonsense. Checking ancient emails to find dates etc reminded me of a huge list of other stuff we were working on back then that *still* needs finishing! And the early pinned objects stuff... and private methods! And...!) Oh-ho - and as an amusing aside, an email from Eliot of that sort of date was pointing out how amazing it was to be able to get a 30Gb disk for a mere $190; you can get 10TB now.
I definitely agree that for an awful lot of cases using a good ffi is better nowadays. It does have some costs though; but in some case I think those are costs we should be pleased to pay. For example, a number of current plugins would be better replaced with ffi classes that actually work with the OS facilities rather than trying to work around them. The FilePlugin comes readily to mind. How much nicer it would be to handle files with direct ffi calls to OS routines and have the complicated stuff in Smalltalk bridge classes. Hmm, now where did we have this in the past; oh yes, in the BrouHaHa images 30-some years ago.
Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.
+1
Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.
+1000
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: DCVP: Destroy another Computer Via Phone-link
Hi
On 25.05.2020, at 05:35, Eliot Miranda eliot.miranda@gmail.com wrote:
Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.
+1
Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.
+1000
One thing tho.
This all boils down to "who moves faster".
Here's a story.
When I tried to get the Self VM running again some years ago, one of the agonizing parts was its extensive use of FFI. Why so? Well, the FFI was coded against a mid-1990 X11. Which, in that form did not exist on either Mac nor Linux. Some parts worked, but a lot of things needed attention in _both_ the VM _and_ the Image. In some respect, the "libraries interfaced via FFI" had moved faster than the VM.
Had Self had a Display Plugin just like Squeak, changes would have been much more simple, since the assumptions in the Self world had nod changed at all.
So, if things move slowly in VM/Image land and more quickly in library land, the answer is Plugins. If things move quickly in VM/Image land and are stable in library land, the answer is FFI.
I see this in SqueakSSL very prominently. The stable interface from the Image land helps a lot. It would be a nightmare to have this via FFI. The effect would be that we have to settle on one SSL library on all platforms, and keep that updated and ship it with the VM. That's by the way what the Pharo VM does and I think it is the wrong way for the problem.
Best regards -Tobias
Tobias Pape Das.Linux@gmx.de schrieb am Mo., 25. Mai 2020, 08:26:
This all boils down to "who moves faster".
Here's a story.
When I tried to get the Self VM running again some years ago, one of the agonizing parts was its extensive use of FFI. Why so? Well, the FFI was coded against a mid-1990 X11. Which, in that form did not exist on either Mac nor Linux. Some parts worked, but a lot of things needed attention in _both_ the VM _and_ the Image. In some respect, the "libraries interfaced via FFI" had moved faster than the VM.
Had Self had a Display Plugin just like Squeak, changes would have been much more simple, since the assumptions in the Self world had nod changed at all.
So, if things move slowly in VM/Image land and more quickly in library land, the answer is Plugins. If things move quickly in VM/Image land and are stable in library land, the answer is FFI.
I see this in SqueakSSL very prominently. The stable interface from the Image land helps a lot. It would be a nightmare to have this via FFI. The effect would be that we have to settle on one SSL library on all platforms, and keep that updated and ship it with the VM. That's by the way what the Pharo VM does and I think it is the wrong way for the problem.
How does using plugins solve this problem?
If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
Do you mean you can hot-fix a plugin underneath the image, so you can run ancient unmodified images on a newer platform/library base? Then maybe we do not really want plugins, but proper modules (Smalltalk shared libraries/image fragments/image components/parcels/...).
On 25.05.2020, at 13:34, Jakob Reschke forums.jakob@resfarm.de wrote:
Tobias Pape Das.Linux@gmx.de schrieb am Mo., 25. Mai 2020, 08:26:
This all boils down to "who moves faster".
Here's a story.
When I tried to get the Self VM running again some years ago, one of the agonizing parts was its extensive use of FFI. Why so? Well, the FFI was coded against a mid-1990 X11. Which, in that form did not exist on either Mac nor Linux. Some parts worked, but a lot of things needed attention in _both_ the VM _and_ the Image. In some respect, the "libraries interfaced via FFI" had moved faster than the VM.
Had Self had a Display Plugin just like Squeak, changes would have been much more simple, since the assumptions in the Self world had nod changed at all.
So, if things move slowly in VM/Image land and more quickly in library land, the answer is Plugins. If things move quickly in VM/Image land and are stable in library land, the answer is FFI.
I see this in SqueakSSL very prominently. The stable interface from the Image land helps a lot. It would be a nightmare to have this via FFI. The effect would be that we have to settle on one SSL library on all platforms, and keep that updated and ship it with the VM. That's by the way what the Pharo VM does and I think it is the wrong way for the problem.
How does using plugins solve this problem?
If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
Do you mean you can hot-fix a plugin underneath the image, so you can run ancient unmodified images on a newer platform/library base? Then maybe we do not really want plugins, but proper modules (Smalltalk shared libraries/image fragments/image components/parcels/...).
Rather the latter. I wouldn't call it hot-fix but rather evolve stuff as they change. That's how we do with Display and SqueakSSL. You don't need to touch the image, but you could. I don't understand how "proper modules" in whatever sense would fix that.
:)
Best regards -Tobias
Tobias Pape Das.Linux@gmx.de schrieb am Mo., 25. Mai 2020, 14:03:
On 25.05.2020, at 13:34, Jakob Reschke forums.jakob@resfarm.de wrote:
Tobias Pape Das.Linux@gmx.de schrieb am Mo., 25. Mai 2020, 08:26:
This all boils down to "who moves faster".
Here's a story.
When I tried to get the Self VM running again some years ago, one of the
agonizing parts was its
extensive use of FFI. Why so? Well, the FFI was coded against a mid-1990
X11. Which, in that form
did not exist on either Mac nor Linux. Some parts worked, but a lot of
things needed attention
in _both_ the VM _and_ the Image. In some respect, the "libraries
interfaced via FFI" had moved
faster than the VM.
Had Self had a Display Plugin just like Squeak, changes would have been
much more simple, since the
assumptions in the Self world had nod changed at all.
So, if things move slowly in VM/Image land and more quickly in library
land, the answer is Plugins.
If things move quickly in VM/Image land and are stable in library land,
the answer is FFI.
I see this in SqueakSSL very prominently. The stable interface from the
Image land helps a lot.
It would be a nightmare to have this via FFI. The effect would be that
we have to
settle on one SSL library on all platforms, and keep that updated and
ship it with the VM.
That's by the way what the Pharo VM does and I think it is the wrong way
for the problem.
How does using plugins solve this problem?
If a library breaks compatibility, you would have to update the plugin
and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
Do you mean you can hot-fix a plugin underneath the image, so you can
run ancient unmodified images on a newer platform/library base? Then maybe we do not really want plugins, but proper modules (Smalltalk shared libraries/image fragments/image components/parcels/...).
Rather the latter. I wouldn't call it hot-fix but rather evolve stuff as they change. That's how we do with Display and SqueakSSL. You don't need to touch the image, but you could. I don't understand how "proper modules" in whatever sense would fix that.
:)
Just to follow up on my modules remark: I meant to split the object memory, or at least the class library, in multiple files, so you can swap one out for another version if needed, even without needing to run the Smalltalk system. Like a Smalltalk shared library. Then you could leave the main part of the image with your work environment untouched, but offline-replace some subystem like the Display or encryption. Like with plugins.
Since your problem with Self seems to be that you could not modify the system because you could not run the system in the first place, such splitting does not help by itself. You would still need a working toolset to modify another Smalltalk image or parts thereof.
Text editors and C compilers seem to be mostly available on developer machines, to edit files that don't belong to the text editor or the C compiler itself. :-)
On 25/05/20 5:04 pm, Jakob Reschke wrote:
If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
Libraries like OpenGL are frustrating because of poorly abstracted, low level functions. Trying to stable facade around its warts will not solve the core problem of having to deal with static code and compile cycles.
The frustration is having to regress from live programming into code-compile-link-test cycle in 2020! Given a choice between live programming and backward compatibility, I would choose live programming because I can close any gaps through live coding.
For instance, a few years back I developed live previews for LaTeX without leaving Squeak (Etoys) by using OSProcess to build a LaTeX pipeline that converted code into image as the code was being typed. It is an inefficient design but it took just half a day and has served well over the years.
But this was at process level. If only I could do the same with DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL, extract its instruction set, interface definitions, state variables and calling conventions. Then I can livecode a class and dynamically compile it (now that we have JIT) to call functions DLL without leaving the image. The code may not be 'portable' in traditional sense. But it will be a lot more manageable than static designs.
Regards .. Subbu
On 2020-05-25, at 10:25 AM, K K Subbu kksubbu.ml@gmail.com wrote:
On 25/05/20 5:04 pm, Jakob Reschke wrote:
If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
Libraries like OpenGL are frustrating because of poorly abstracted, low level functions. Trying to stable facade around its warts will not solve the core problem of having to deal with static code and compile cycles.
The frustration is having to regress from live programming into code-compile-link-test cycle in 2020! Given a choice between live programming and backward compatibility, I would choose live programming because I can close any gaps through live coding.
Nicely put.
For instance, a few years back I developed live previews for LaTeX without leaving Squeak (Etoys) by using OSProcess to build a LaTeX pipeline that converted code into image as the code was being typed. It is an inefficient design but it took just half a day and has served well over the years.
But this was at process level. If only I could do the same with DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL, extract its instruction set, interface definitions, state variables and calling conventions. Then I can livecode a class and dynamically compile it (now that we have JIT) to call functions DLL without leaving the image. The code may not be 'portable' in traditional sense. But it will be a lot more manageable than static designs.
VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc. Is that a practical way to do it? Hell of a lot of work to do, certainly.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: QVC: Question Valid Command
On 26/05/20 2:20 am, tim Rowledge wrote:
But this was at process level. If only I could do the same with DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL, extract its instruction set, interface definitions, state variables and calling conventions. Then I can livecode a class and dynamically compile it (now that we have JIT) to call functions DLL without leaving the image. The code may not be 'portable' in traditional sense. But it will be a lot more manageable than static designs.
VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc. Is that a practical way to do it? Hell of a lot of work to do, certainly.
We don't know what source code was used to compile the DLL. Having an inspector for DLLs would be a good start. If we pull out the dynamic link maps into the image, then we can simulate in the image, what VM currently does through dlopen() calls. Like what we do in doPrimitive.
Regards .. Subbu
On 2020-05-26, at 4:20 AM, K K Subbu kksubbu.ml@gmail.com wrote:
On 26/05/20 2:20 am, tim Rowledge wrote:
But this was at process level. If only I could do the same with DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL, extract its instruction set, interface definitions, state variables and calling conventions. Then I can livecode a class and dynamically compile it (now that we have JIT) to call functions DLL without leaving the image. The code may not be 'portable' in traditional sense. But it will be a lot more manageable than static designs.
VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc. Is that a practical way to do it? Hell of a lot of work to do, certainly.
We don't know what source code was used to compile the DLL.
Certainly true in a lot of cases; the idea used to be that you could get a header file that described the interface and so compile appropriate code. I'm sure there are plenty of cases where that is not what actually happens.
Having an inspector for DLLs would be a good start. If we pull out the dynamic link maps into the image, then we can simulate in the image, what VM currently does through dlopen() calls. Like what we do in doPrimitive.
That would be nice. And of course Apple bundles, unix .so's and RISC OS shared libraries. A small matter of programming :-)
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful Latin Phrases:- Magister Mundi sum! = I am the Master of the Universe!
I was hoping getting a solid libCLANG interface would make this easy.
But I haven't been able to get past a problem with some functions returning structs by value and the FFI failing to handle this.
Might be fixed now. Worth a look.
On May 25, 2020, at 1:50 PM, tim Rowledge tim@rowledge.org wrote:
VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc. Is that a practical way to do it? Hell of a lot of work to do, certainly.
On May 26, 2020, at 6:51 PM, Todd Blanchard tblanchard@mac.com wrote:
I was hoping getting a solid libCLANG interface would make this easy.
But I haven't been able to get past a problem with some functions returning structs by value and the FFI failing to handle this.
Bugs were fixed on x86_64 recently and ARMv8 less recently.
There really isn’t any advantage in using a different FFI implementation. We have the capability of producing a correct FFI. What we need is good bug reports and help. What we *dont* need is a different FFI, incoherent or split community effort, etc.
Might be fixed now. Worth a look.
On May 25, 2020, at 1:50 PM, tim Rowledge tim@rowledge.org wrote:
VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc. Is that a practical way to do it? Hell of a lot of work to do, certainly.
Hi Tim,
On May 25, 2020, at 1:50 PM, tim Rowledge tim@rowledge.org wrote:
On 2020-05-25, at 10:25 AM, K K Subbu kksubbu.ml@gmail.com wrote:
On 25/05/20 5:04 pm, Jakob Reschke wrote: If a library breaks compatibility, you would have to update the plugin and maybe think about how to stay compatible with older Squeaks. If it were FFI with a proper facade that finds out how to link to the external library, you would need to update the facade and worry there about staying compatible, but you could do it in Smalltalk. It requires discipline and guidance to only use the facade, of course.
Libraries like OpenGL are frustrating because of poorly abstracted, low level functions. Trying to stable facade around its warts will not solve the core problem of having to deal with static code and compile cycles.
The frustration is having to regress from live programming into code-compile-link-test cycle in 2020! Given a choice between live programming and backward compatibility, I would choose live programming because I can close any gaps through live coding.
Nicely put.
For instance, a few years back I developed live previews for LaTeX without leaving Squeak (Etoys) by using OSProcess to build a LaTeX pipeline that converted code into image as the code was being typed. It is an inefficient design but it took just half a day and has served well over the years.
But this was at process level. If only I could do the same with DLLs. If only Squeak had a tool (ELF reader?) to reify a DLL, extract its instruction set, interface definitions, state variables and calling conventions. Then I can livecode a class and dynamically compile it (now that we have JIT) to call functions DLL without leaving the image. The code may not be 'portable' in traditional sense. But it will be a lot more manageable than static designs.
VW has a parser of C header files that is I suppose the right way to do this sort of thing - you parse the headers (warning - I seem to recall parsing 'windows.h' taking several days in the original VW ddl product) and build proxies of all the types/structs/etc. Is that a practical way to do it? Hell of a lot of work to do, certainly.
Emphatically not. DLLCC’s C parser is a nightmare to maintain, and with a little cunning, essentially unnecessary. See all the conversation around generating a C program which when run outputs information that the image can easily parse, and as a result (re)populate a shared pool with relevant platform specifics. We have this in FFISharedPool (by Shaping IIRC, after an idea/suggestion of mine, implemented as a prototype before I left Cincom, and as a part of UFFI by Mariano Martinez Peck, following discussion a few years back).
Generating a program has several advantages
- no need to implement any part of the C toolchain (doesn’t reinvent the wheel)
- uses the platform’s information more-or-less directly, rather than trying to mimic it
- the binaries can be shared between developers (a library of them?) and deployed with cross-platform apps
- only the FFI api author needs a C development toolset
So what the FFI author does is define a shared pool which has platform-specific metadata, specifically the necessary include files and version selection defines for each relevant platform. The FFI shared pool machinery takes care of generating the program when necessary (only when authoring an FFI), and running it when necessary (only once for a single-platform app, only when launching on a different platform to the one snapshotted on)
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: QVC: Question Valid Command
On 2020-05-26, at 8:10 PM, Eliot Miranda eliot.miranda@gmail.com wrote: DLLCC’s C parser is a nightmare to maintain,
I certainly remember *that*. Poor Marcel Schelvis practically went mad working on it.
So what the FFI author does is define a shared pool which has platform-specific metadata, specifically the necessary include files and version selection defines for each relevant platform. The FFI shared pool machinery takes care of generating the program when necessary (only when authoring an FFI), and running it when necessary (only once for a single-platform app, only when launching on a different platform to the one snapshotted on)
Sounds good to me. Beats parsing secondhand and possibly inaccurate information intended for an inferior language
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Fractured Idiom:- MERCI RIEN - Thanks for nothin'.
Le mer. 27 mai 2020 à 07:09, tim Rowledge tim@rowledge.org a écrit :
On 2020-05-26, at 8:10 PM, Eliot Miranda eliot.miranda@gmail.com
wrote:
DLLCC’s C parser is a nightmare to maintain,
I certainly remember *that*. Poor Marcel Schelvis practically went mad working on it.
If you try putting C++ syntax on a postcard, you probably gonna beat the
number of folds you can perform on a sheet before it fits in any mailing box.
So what the FFI author does is define a shared pool which has platform-specific metadata, specifically the necessary include files and version selection defines for each relevant platform. The FFI shared pool machinery takes care of generating the program when necessary (only when authoring an FFI), and running it when necessary (only once for a single-platform app, only when launching on a different platform to the one snapshotted on)
Sounds good to me. Beats parsing secondhand and possibly inaccurate information intended for an inferior language
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Fractured Idiom:- MERCI RIEN - Thanks for nothin'.
On 2020-05-26, at 10:29 PM, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:
Le mer. 27 mai 2020 à 07:09, tim Rowledge tim@rowledge.org a écrit :
On 2020-05-26, at 8:10 PM, Eliot Miranda eliot.miranda@gmail.com wrote: DLLCC’s C parser is a nightmare to maintain,
I certainly remember *that*. Poor Marcel Schelvis practically went mad working on it.
If you try putting C++ syntax on a postcard, you probably gonna beat the number of folds you can perform on a sheet before it fits in any mailing box.
I think that 'postcard' would end up the size of one of those giant roadside posters that require guys on a crane to install.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: DNPG: Do Not Pass Go
On Sun, May 24, 2020 at 23:14, tim Rowledge tim@rowledge.org wrote:
On 2020-05-24, at 3:57 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Plugins emerged from the lack of an FFI, later a deficient FFI (slow, non-reentrant, no callbacks), and the need for security, at a time when graphics co-processing was very rare and not at all commodity technology. The world looks different now. We should evolve to match the new environment. I agree that security is very important. But I do not believe that security implies no FFI. I think we can have both security and the flexibility, development speed and architectural efficiency of an FFI-based implementation.
The VM plugins were an attempt to get away from a few problems that plagued VM development at the end of the last millennium (and I don't mean March this year) with things like a limited number of prim table entires, the lack of flexibility a static table engendered, the time it took to compile an entire VM for a tiny change/test cycle etc (IIRC of the order of half an hour back then). The original external plugins stuff meant you could get to outside code via a name system that didn't screw the prim table (and I'd still love to find away to get rid of it entirely, but dream on) and that let you build/load/try/unload/rebuild quickly. We were also starting the beginnings of the ffi stuff around then. Andy Greenberg came up with the SmartSyntaxPlugin stuff, for example, at the end of '99. Apparently I sent Dan/Andreas a final pluginized linux vm late April 2000.
It's worth remembering just how awful Windows & Mac OS were back then as part of the wider scenario - non-preemptive processscheduling, weird callback based event reading, general nonsense. Checking ancient emails to find dates etc reminded me of a huge list of other stuff we were working on back then that *still* needs finishing! And the early pinned objects stuff... and private methods! And...!) Oh-ho - and as an amusing aside, an email from Eliot of that sort of date was pointing out how amazing it was to be able to get a 30Gb disk for a mere $190; you can get 10TB now.
I definitely agree that for an awful lot of cases using a good ffi is better nowadays. It does have some costs though; but in some case I think those are costs we should be pleased to pay. For example, a number of current plugins would be better replaced with ffi classes that actually work with the OS facilities rather than trying to work around them. The FilePlugin comes readily to mind. How much nicer it would be to handle files with direct ffi calls to OS routines and have the complicated stuff in Smalltalk bridge classes. Hmm, now where did we have this in the past; oh yes, in the BrouHaHa images 30-some years ago.
Moving to ffi-only would allow removing a fair bit of cruft from the VM, though there's our old friend 'backward compatibility' to argue with. It would mean some rewriting (mostly simplification I think?) of existing plugins code. And what to do about stuff that we pretty much always compile as internal plugins? Rewrite as external libraries and pay the (hopefully small) performance cost, or rewrite as appropriate to drop any pretence of being able to be external? SoundPlugin should absolutely be replaced by a bridge class and ffi calls to the OS. LargeInterger? Probably much better to pull properly into the vm core. MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.
Strip the VM. It should have just the bare abilities to load, run and save images, and call the OS for anything to do with i/o etc. That'll be one meeeeeeeeeeeeeeeeellion dollars please.
How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?
I feel like a special SharedObjectLibPlugin super class could generate the lib and the FFI calls to use it. Then the plugin slang could still be managed in-image, yet use FFI. Is it so?
Another use case I can think of for use of FFI, is non-blocking socket code, NIO Sockets.
Kindly, Robert
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: DCVP: Destroy another Computer Via Phone-link
OK, I'll try to answer some of the thoughts on this -
On 2020-05-24, at 11:07 PM, Robert robert.withers@pm.me wrote:
How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?
Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.
On 2020-05-24, at 11:26 PM, Tobias Pape Das.Linux@gmx.de wrote: The effect would be that we have to settle on one SSL library on all platforms, and keep that updated and ship it with the VM. That's by the way what the Pharo VM does and I think it is the wrong way for the problem.
Actually no, that isn't how you *have* to do it. You can have interfaces from the image to many different libraries; hence the mentions of 'bridge classes' in my message. It's not much different to how we currently choose the platform appropriate class of file directory on startup.
When the interface to a subsystem changes you have to make corresponding changes *somewhere*. With current plugins we have to rewrite the relevant parts, often the hand-written C-shim stuff. In one sense that is a virtue since we can do that with 'normal' tools like edlin, and enjoy all the fun of that world of high quality IDEs (sarcasm? Moi?). If we used bridge classes to ffi calls direct to the external system then we would have to do some work in the image; now of course in general we'd all prefer that but I can imagine cases where the changed interface prevents us running the image to write the code to handle the changed interface. This is where having a way to make the system load code at startup helps.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Eagles may soar, but weasels aren't sucked into jet engines.
On 25.05.2020, at 18:46, tim Rowledge tim@rowledge.org wrote:
OK, I'll try to answer some of the thoughts on this -
On 2020-05-24, at 11:07 PM, Robert robert.withers@pm.me wrote:
How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?
Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.
On 2020-05-24, at 11:26 PM, Tobias Pape Das.Linux@gmx.de wrote: The effect would be that we have to settle on one SSL library on all platforms, and keep that updated and ship it with the VM. That's by the way what the Pharo VM does and I think it is the wrong way for the problem.
Actually no, that isn't how you *have* to do it. You can have interfaces from the image to many different libraries; hence the mentions of 'bridge classes' in my message. It's not much different to how we currently choose the platform appropriate class of file directory on startup.
Per se, true. But try discriminating openssl 0.9.6, 1.0.1, and 1.1.1. It's a nightmare. The overlay somewhat works, but even that is not so easy.
It's just that I've seen the bridge-class stuff made things hard for me in Self. But that might just be the long time gap where X11 evolved and the Self part did not. Problem was I had no tools because I broke my tools with my tools (so to speak, because no UI, no way of fixing the UI properly).
When the interface to a subsystem changes you have to make corresponding changes *somewhere*. With current plugins we have to rewrite the relevant parts, often the hand-written C-shim stuff. In one sense that is a virtue since we can do that with 'normal' tools like edlin, and enjoy all the fun of that world of high quality IDEs (sarcasm? Moi?).
Not that I like digging in C more than using Squeak proper...
If we used bridge classes to ffi calls direct to the external system then we would have to do some work in the image; now of course in general we'd all prefer that but I can imagine cases where the changed interface prevents us running the image to write the code to handle the changed interface. This is where having a way to make the system load code at startup helps.
Again, FFI is fine. My point is rather that the pace of change can indicate whether a plugin is better or FFI.
Also, the other VM-implementations can provide Plugins much more easily than FFI. I would argue this is true for SqueakJS, RSqueak, JSqueak/Potato, and to an extend, TruffleSqueak. These systems do not necessarily provide access to C libraries and reduce the need for in-image changes for each VM.
TL;DR: FFI good, Plugins also good.
just my 2ct.
Best regards -Tobias
On Mon, May 25, 2020 at 07:16:02PM +0200, Tobias Pape wrote:
TL;DR: FFI good, Plugins also good.
just my 2ct.
Best regards -Tobias
Well said. I would only add that Juan Vuletich has written a nice, balanced overview of the use of FFI and VM plugins here:
https://lists.cuis.st/mailman/archives/cuis-dev/2020-May/001736.html
Dave
wrt the speed of calls to plugins, I wrote that mechanism a few years back (um, actually 15 years apparently) and the actual routine call should be exactly as fast as a number primitive - the code address is in the cache tables. Hmm.. does cog inline cache stuff affect that? I don't think so but could be wrong.
The only slow-down likely would be the potentially longer-winded routines to get stuff of the stack/instvars and put answers back. If one were to postulate changing to only internal plugins (to provide some modicum of modularity and maintain the named prim interface that was a key original driver) and having all external code be ffi called, then I rather think we could improve the get/put code.
On 2020-05-27, at 3:28 PM, David T. Lewis lewis@mail.msen.com wrote:
https://lists.cuis.st/mailman/archives/cuis-dev/2020-May/001736.html
Yup; a very good summary.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim A conscience is what hurts when all your other parts feel so good.
OMG, I have had a huge brain fart. I totally forgot that plugins are built *as* shared object Libs.
So our choice to invoke said SOLib through named primitives are just as workable as somehow invoking FFI on them. What would that look like 👀? Is there an example for calling FFI, for one of the Crypto plugins? Let’s evaluate.
Kindly, Robert
On Mon, May 25, 2020 at 12:46, tim Rowledge tim@rowledge.org wrote:
OK, I'll try to answer some of the thoughts on this -
On 2020-05-24, at 11:07 PM, Robert robert.withers@pm.me wrote:
How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?
Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.
On 2020-05-24, at 11:26 PM, Tobias Pape Das.Linux@gmx.de wrote: The effect would be that we have to settle on one SSL library on all platforms, and keep that updated and ship it with the VM. That's by the way what the Pharo VM does and I think it is the wrong way for the problem.
Actually no, that isn't how you *have* to do it. You can have interfaces from the image to many different libraries; hence the mentions of 'bridge classes' in my message. It's not much different to how we currently choose the platform appropriate class of file directory on startup.
When the interface to a subsystem changes you have to make corresponding changes *somewhere*. With current plugins we have to rewrite the relevant parts, often the hand-written C-shim stuff. In one sense that is a virtue since we can do that with 'normal' tools like edlin, and enjoy all the fun of that world of high quality IDEs (sarcasm? Moi?). If we used bridge classes to ffi calls direct to the external system then we would have to do some work in the image; now of course in general we'd all prefer that but I can imagine cases where the changed interface prevents us running the image to write the code to handle the changed interface. This is where having a way to make the system load code at startup helps.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Eagles may soar, but weasels aren't sucked into jet engines.
On Mon, May 25, 2020 at 12:46, tim Rowledge tim@rowledge.org wrote:
OK, I'll try to answer some of the thoughts on this -
On 2020-05-24, at 11:07 PM, Robert robert.withers@pm.me wrote:
How about the CryptographyPlugins? What would switching to FFI look like? I feel like we still need custom C code implementing the algorithms.
Is it that we would have shared object libraries (.so/DLLs), generated from slang, but not creating plugins, just generating .so libs and the FFI calls to use them?
Right now (assuming I'm remembering it all correctly and with the proviso that I haven't written a new plugin in several years) an external plugin is compiled from the assorted code required (some plugins are solely slang generated code, some are solely hand-written C, some are slang + C + calls to other libraries and so on) and a platform specific form of shared library is made. Are they in a form that is callable via 'normal' ffi calls? Dunno, but clearly they *could* be made that way with not very much change.
Tim, does this mean that the plugin generation and linking as appropriate continues a pace, while the in-image named prims table switches all entries to FFI calls? CryptographyPlugins have the advantage of slang generated only code.
Would it be possible to take advantage of a GPU??
K, r
On Sun, 24 May 2020, tim Rowledge wrote:
MiscPlugins? Wouldn't be surprised if they could all be dropped and the cog'd versions would run as fast.
Below is a benchmark measuring the primitive to calculate string and byte array hashes. The in-image variant is fully JIT-friendly: AFAIK all its message sends have jitted implementations (#bitAnd:, #size, #to:do:, #+, #basicAt:, #hashMultiply).
| b | b := ByteArray new: 10000. {[ b hashWithInitialHash: 0 ] benchFor: 1 seconds. [ String stringHash: b initialHash: 0 ] benchFor: 1 seconds }. #('88,500 per second. 11.3 microseconds per run. 0 % GC time.' '11,500 per second. 86.9 microseconds per run. 0 % GC time.')"
In this case, the primitive version is 7.7x faster on my machine.
The actual performance difference depends on the size of the collection. For sizes smaller than 4, the primitive version is slower. That could probably be improved by making it a numberered primitive and providing a jitted version of the primitive; something that was done with #compare:with:collated: (see primitive 158).
At this point we clearly can't drop those primitives. Some of them is probably worth to add to the VM directly, others may not be necessary at all in the future.
Levente
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: DCVP: Destroy another Computer Via Phone-link
vm-dev@lists.squeakfoundation.org
-
David T. Lewis -
Eliot Miranda -
Jakob Reschke -
K K Subbu -
Levente Uzonyi -
Nicolas Cellier -
Robert -
tim Rowledge -
Tobias Pape -
Todd Blanchard