Does anyone use the plugin mechanism to load libraries that are not plugins on Unix? Or to override just one or two of the installed ones from a user directory?
I'm thinking of simplifying the search strategy (which is indeed broken w.r.t. overriding installed plugins as Subbu points out) along with all the junk related to probing for a zillion prefixes and suffixes. Unlike libtool, CMake manages to build loadable modules with predictable 'lib*.so' names regardless of the platform. That, combined with a launch script that can add a -plugins option to the VM args, suggests it ought to be possible to find the plugin precisely on the first attempt, without having to search at all.
The advantage is vastly simpler logic that is completely predictable.
The disadvantage is that it will not be possible to subvert the plugin mechanism to load system libraries, and it will not be possible to override the installed plugins with a single plugin built in a different directory. Would either of these be a noticeable loss?
Cheers, Ian
Would this affect the FFI and its attempts to load platform libraries?
Cheers, - Andreas
Ian Piumarta wrote:
Does anyone use the plugin mechanism to load libraries that are not plugins on Unix? Or to override just one or two of the installed ones from a user directory?
I'm thinking of simplifying the search strategy (which is indeed broken w.r.t. overriding installed plugins as Subbu points out) along with all the junk related to probing for a zillion prefixes and suffixes. Unlike libtool, CMake manages to build loadable modules with predictable 'lib*.so' names regardless of the platform. That, combined with a launch script that can add a -plugins option to the VM args, suggests it ought to be possible to find the plugin precisely on the first attempt, without having to search at all.
The advantage is vastly simpler logic that is completely predictable.
The disadvantage is that it will not be possible to subvert the plugin mechanism to load system libraries, and it will not be possible to override the installed plugins with a single plugin built in a different directory. Would either of these be a noticeable loss?
Cheers, Ian
On Aug 26, 2009, at 8:02 PM, Andreas Raab wrote:
Would this affect the FFI and its attempts to load platform libraries?
It would continue to find anything that dlopen() would find; i.e., anything with the platform's expected prefix and suffix in the standard shared library locations including whatever influence environment variables might have on the search.
In other words, I am suggesting:
1. The SQUEAK_PLUGINS or -plugins location (which could be multiple paths separated by ':'s) with 'lib' prefix and '.so' suffix. This finds the VM plugins on all platforms regardless of local library naming conventions and permits non-installed libraries to override installed ones.
2. The default places searched by dlopen() with the platform's default library prefix and suffix added, unless there is a / in the name in which case I'd pass it verbatim. This finds the FFI libraries (which AFAICT don't tend to have prefix/suffix in their module name in FFI declarations).
Anyone wanting to find platform libraries in non-standard places (e.g., X11 on many non-Linux platforms) would have to provide a full path to the library or set LD_LIBRARY_PATH (or whatever) as required.
John's report makes me think the above is almost reasonable. :)
Cheers, Ian
Cheers,
- Andreas
Ian Piumarta wrote:
Does anyone use the plugin mechanism to load libraries that are not plugins on Unix? Or to override just one or two of the installed ones from a user directory? I'm thinking of simplifying the search strategy (which is indeed broken w.r.t. overriding installed plugins as Subbu points out) along with all the junk related to probing for a zillion prefixes and suffixes. Unlike libtool, CMake manages to build loadable modules with predictable 'lib*.so' names regardless of the platform. That, combined with a launch script that can add a - plugins option to the VM args, suggests it ought to be possible to find the plugin precisely on the first attempt, without having to search at all. The advantage is vastly simpler logic that is completely predictable. The disadvantage is that it will not be possible to subvert the plugin mechanism to load system libraries, and it will not be possible to override the installed plugins with a single plugin built in a different directory. Would either of these be a noticeable loss? Cheers, Ian
On Thursday 27 Aug 2009 9:15:39 am Ian Piumarta wrote:
Anyone wanting to find platform libraries in non-standard places (e.g., X11 on many non-Linux platforms) would have to provide a full path to the library or set LD_LIBRARY_PATH (or whatever) as required.
John's report makes me think the above is almost reasonable. :)
"there must be a simpler way" .. Dan Ingalls (talk on lively kernel)
Subbu
Ian Piumarta wrote:
Would this affect the FFI and its attempts to load platform libraries?
It would continue to find anything that dlopen() would find; i.e., anything with the platform's expected prefix and suffix in the standard shared library locations including whatever influence environment variables might have on the search.
But it wouldn't find it in any of the "additional" locations that you were looking at, right? It's probably okay (most Unix users know that they have to set LD_LIBRARY_PATH) but I'm still a little concerned that this may break apps if people switch from one VM to the next. Perhaps leave it as an option for the FFI and drop it for good on the 4.0 switch? (changes in major versions are just easier to explain than the change from 3.10-6 to 3.10-7 ;-)
It's your choice of course.
Cheers, - Andreas
In other words, I am suggesting:
- The SQUEAK_PLUGINS or -plugins location (which could be multiple
paths separated by ':'s) with 'lib' prefix and '.so' suffix. This finds the VM plugins on all platforms regardless of local library naming conventions and permits non-installed libraries to override installed ones.
- The default places searched by dlopen() with the platform's default
library prefix and suffix added, unless there is a / in the name in which case I'd pass it verbatim. This finds the FFI libraries (which AFAICT don't tend to have prefix/suffix in their module name in FFI declarations).
Anyone wanting to find platform libraries in non-standard places (e.g., X11 on many non-Linux platforms) would have to provide a full path to the library or set LD_LIBRARY_PATH (or whatever) as required.
John's report makes me think the above is almost reasonable. :)
Cheers, Ian
Cheers,
- Andreas
Ian Piumarta wrote:
Does anyone use the plugin mechanism to load libraries that are not plugins on Unix? Or to override just one or two of the installed ones from a user directory? I'm thinking of simplifying the search strategy (which is indeed broken w.r.t. overriding installed plugins as Subbu points out) along with all the junk related to probing for a zillion prefixes and suffixes. Unlike libtool, CMake manages to build loadable modules with predictable 'lib*.so' names regardless of the platform. That, combined with a launch script that can add a -plugins option to the VM args, suggests it ought to be possible to find the plugin precisely on the first attempt, without having to search at all. The advantage is vastly simpler logic that is completely predictable. The disadvantage is that it will not be possible to subvert the plugin mechanism to load system libraries, and it will not be possible to override the installed plugins with a single plugin built in a different directory. Would either of these be a noticeable loss? Cheers, Ian
On Thursday 27 Aug 2009 2:15:19 pm Andreas Raab wrote:
It would continue to find anything that dlopen() would find; i.e., anything with the platform's expected prefix and suffix in the standard shared library locations including whatever influence environment variables might have on the search.
But it wouldn't find it in any of the "additional" locations that you were looking at, right?
if it did, wouldn't it be a bug? .. Subbu
Andreas Raab wrote:
Ian Piumarta wrote:
Would this affect the FFI and its attempts to load platform libraries?
It would continue to find anything that dlopen() would find; i.e., anything with the platform's expected prefix and suffix in the standard shared library locations including whatever influence environment variables might have on the search.
But it wouldn't find it in any of the "additional" locations that you were looking at, right? It's probably okay (most Unix users know that they have to set LD_LIBRARY_PATH) but I'm still a little concerned that this may break apps if people switch from one VM to the next. Perhaps leave it as an option for the FFI and drop it for good on the 4.0 switch? (changes in major versions are just easier to explain than the change from 3.10-6 to 3.10-7 ;-)
On platforms using ELF executables you can embed paths for the run time linker into the exe itself so each vm (and plugin) can be built to look in its own specific places.
On Aug 27, 2009, at 5:21 AM, Douglas Brebner wrote:
On platforms using ELF executables you can embed paths for the run time linker into the exe itself so each vm (and plugin) can be built to look in its own specific places.
The ideal would be to set an RPATH on libSqueakFFIPrims.so that contains the complete list of likely places to search for FFI libraries, but not all platforms are ELF and it appears that RPATH is not inherited by the main program (where the dlopen() call is made) from dynamically-loaded libraries even when control has passed through such a library to reach the dlopen() call. (This is arguably a huge bug in the semantics of C.)
Part of the complication arises because loading a primitive plugin and loading a FFI library are conflated in the support code but should behave differently.
There is a distinction within the Interpreter between loading a "plugin module" for primitives and loading a "library module" for FFI callout. Unfortunately the distinction is destroyed by the time we reach ioLoadModule and ioFindExternalFunction in the support code.
However, there are two functions in the vm proxy structure used exclusively by FFI to find libraries and symbols that appear to come before the plugin/library distinction is destroyed. It would be possible to wrap those trivially such that the platform support can tell if the ioLoadModule came from a primitive or from the FFI. The former would look for a piece of VM (by adding plugindir/prefix and .suffix to the module name) and the latter would look for a library (by doing whatever dlopen() does by default: FFI library not found? Go RTFM page of dlopen, rtld, dyld, whatever...).
Cheers, Ian
On Wed, Aug 26, 2009 at 07:16:42PM -0700, Ian Piumarta wrote:
Does anyone use the plugin mechanism to load libraries that are not plugins on Unix? Or to override just one or two of the installed ones from a user directory?
I'm thinking of simplifying the search strategy (which is indeed broken w.r.t. overriding installed plugins as Subbu points out) along with all the junk related to probing for a zillion prefixes and suffixes. Unlike libtool, CMake manages to build loadable modules with predictable 'lib*.so' names regardless of the platform. That, combined with a launch script that can add a -plugins option to the VM args, suggests it ought to be possible to find the plugin precisely on the first attempt, without having to search at all.
The advantage is vastly simpler logic that is completely predictable.
The disadvantage is that it will not be possible to subvert the plugin mechanism to load system libraries, and it will not be possible to override the installed plugins with a single plugin built in a different directory. Would either of these be a noticeable loss?
Predictable and easy to understand is more important than convenience, so it sounds like a net win to me.
I like being able to work on a plugin that I can build and run from a local build directory (cd ./build; make; ./squeak myimage) and have it find the plugin in the ./build directory. I also like to be able to run my ./build/squeak under gdb for debugging the plugin. It's nice if that mechanism continues to work but not essential.
Dave
I've a flag in the mac carbon vm that turns off oh 90% of the searching, that is ON by default, people haven't complained. Your milage might be different for Linux users.
SqueakPluginsBuiltInOrLocalOnly
when set to true makes the lookup logic only consider unix libraries or os-x bundles in the ./Plugins folder in the same directory as the app, or the application Resources folder, or as a foo.framework in the /System/Library/Frameworks/.
To turn this feature off and then consider the other 80+ choices of library locations and library names set SqueakPluginsBuiltInOrLocalOnly to false.Turning the feature on avoids a couple of hundred stat calls at startup time as the VM hunts for external versions of internal plugins. It was added to reduce overhead and help the macIntel migration were we provided new universal binaries that live in the application Resources folder since we cann't load powerpc plugins on macintel machines if we found them first in the ./Pugins folder
We skip looking for
static char *prefixes[]= { "", "lib", 0 }; static char *suffixes[]= { "", ".so", ".dylib",0 };
and for local or built in does the vm directory path (which is the os- x bundle resource folder), then ./Plugins
if ( (handle= tryLoading( vmDirPath, pluginName)) || (handle= tryLoading( pluginDirPath, pluginName))
and skips || (handle= tryLoading( "./", pluginName)) || (handle= tryLoadingPath("SQUEAK_PLUGIN_PATH", pluginName)) || (handle= tryLoading( VM_LIBDIR"/", pluginName)) || (handle= tryLoadingPath("LD_LIBRARY_PATH", pluginName)) || (handle= tryLoading( "", pluginName)) # if defined(VM_X11DIR) || (handle= tryLoading(VM_X11DIR"/", pluginName)) # endif )
A special case to this is a bit more hunting if the plugin is *.framework since we then consider the vmDir, the puginDirpath, and the systemFolder
Otherwise it would also hunt in "/CoreServices.framework/Frameworks", "/ApplicationServices.framework/Frameworks", "/Carbon.framework/Frameworks",
On 26-Aug-09, at 7:16 PM, Ian Piumarta wrote:
Does anyone use the plugin mechanism to load libraries that are not plugins on Unix? Or to override just one or two of the installed ones from a user directory?
I'm thinking of simplifying the search strategy (which is indeed broken w.r.t. overriding installed plugins as Subbu points out) along with all the junk related to probing for a zillion prefixes and suffixes. Unlike libtool, CMake manages to build loadable modules with predictable 'lib*.so' names regardless of the platform. That, combined with a launch script that can add a - plugins option to the VM args, suggests it ought to be possible to find the plugin precisely on the first attempt, without having to search at all.
The advantage is vastly simpler logic that is completely predictable.
The disadvantage is that it will not be possible to subvert the plugin mechanism to load system libraries, and it will not be possible to override the installed plugins with a single plugin built in a different directory. Would either of these be a noticeable loss?
Cheers, Ian
-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Twitter: squeaker68882 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================
On Thu, Aug 27, 2009 at 4:16 AM, Ian Piumartapiumarta@speakeasy.net wrote:
Does anyone use the plugin mechanism to load libraries that are not plugins on Unix? Or to override just one or two of the installed ones from a user directory?
I'm thinking of simplifying the search strategy (which is indeed broken w.r.t. overriding installed plugins as Subbu points out) along with all the junk related to probing for a zillion prefixes and suffixes. Unlike libtool, CMake manages to build loadable modules with predictable 'lib*.so' names regardless of the platform. That, combined with a launch script that can add a -plugins option to the VM args, suggests it ought to be possible to find the plugin precisely on the first attempt, without having to search at all.
The advantage is vastly simpler logic that is completely predictable.
Agreed.
The disadvantage is that it will not be possible to subvert the plugin mechanism to load system libraries, and it will not be possible to override the installed plugins with a single plugin built in a different directory. Would either of these be a noticeable loss?
Nope. Go for it!
- Andrew
On 27.08.2009, at 10:36, Andrew Gaylard wrote:
On Thu, Aug 27, 2009 at 4:16 AM, Ian Piumartapiumarta@speakeasy.net wrote:
Does anyone use the plugin mechanism to load libraries that are not plugins on Unix? Or to override just one or two of the installed ones from a user directory?
I'm thinking of simplifying the search strategy (which is indeed broken w.r.t. overriding installed plugins as Subbu points out) along with all the junk related to probing for a zillion prefixes and suffixes. Unlike libtool, CMake manages to build loadable modules with predictable 'lib*.so' names regardless of the platform. That, combined with a launch script that can add a -plugins option to the VM args, suggests it ought to be possible to find the plugin precisely on the first attempt, without having to search at all.
The advantage is vastly simpler logic that is completely predictable.
Agreed.
The disadvantage is that it will not be possible to subvert the plugin mechanism to load system libraries, and it will not be possible to override the installed plugins with a single plugin built in a different directory. Would either of these be a noticeable loss?
Nope. Go for it!
- Andrew
+1
As for debugging, what I usually do is symlink the plugin from my build dir to the default plugin dir.
Out of curiosity, which other extensions than .so are in use currently?
- Bert -
On Thu, Aug 27, 2009 at 2:10 AM, Bert Freudenberg bert@freudenbergs.dewrote:
On 27.08.2009, at 10:36, Andrew Gaylard wrote:
On Thu, Aug 27, 2009 at 4:16 AM, Ian Piumartapiumarta@speakeasy.net
wrote:
Does anyone use the plugin mechanism to load libraries that are not plugins on Unix? Or to override just one or two of the installed ones from a user directory?
I'm thinking of simplifying the search strategy (which is indeed broken w.r.t. overriding installed plugins as Subbu points out) along with all the junk related to probing for a zillion prefixes and suffixes. Unlike libtool, CMake manages to build loadable modules with predictable 'lib*.so' names regardless of the platform. That, combined with a launch script that can add a -plugins option to the VM args, suggests it ought to be possible to find the plugin precisely on the first attempt, without having to search at all.
The advantage is vastly simpler logic that is completely predictable.
Agreed.
The disadvantage is that it will not be possible to subvert the plugin
mechanism to load system libraries, and it will not be possible to override the installed plugins with a single plugin built in a different directory. Would either of these be a noticeable loss?
Nope. Go for it!
- Andrew
+1
As for debugging, what I usually do is symlink the plugin from my build dir to the default plugin dir.
Out of curiosity, which other extensions than .so are in use currently?
HPUX uses .sl & Mac OS X uses .dylib
- Bert -
On 27.08.2009, at 17:59, Eliot Miranda wrote:
On Thu, Aug 27, 2009 at 2:10 AM, Bert Freudenberg <bert@freudenbergs.de
wrote:
Out of curiosity, which other extensions than .so are in use currently?
HPUX uses .sl & Mac OS X uses .dylib
Hmm. So maybe cmake could figure this out?
- Bert -
On Aug 27, 2009, at 9:32 AM, Bert Freudenberg wrote:
HPUX uses .sl & Mac OS X uses .dylib
Hmm. So maybe cmake could figure this out?
It does. add_library(SHARED ...) uses the platform's native format. add_library(MODULE ...) uses 'lib' in front and '.so' in back. There are variables to tell you what the native prefix and sufix are for libraries.
The biggest issue was libtool using some totally random combination of name, lib (or not), .so or .dylib (or not), version number (in one two or three parts, correct or all zeros, placed before or after the .so and maybe separated by a -, or maybe not) each according to the ls bit in /dev/random multiplied by the phase of the moon. Buried somewhere in a .lobscure file someplace might be a clue as to what it decided to do.
Cheers, Ian (a cmaker who is never going back to autohell)
On Thu, Aug 27, 2009 at 5:59 PM, Eliot Mirandaeliot.miranda@gmail.com wrote:
Out of curiosity, which other extensions than .so are in use currently?
HPUX uses .sl & Mac OS X uses .dylib
Yeah, but let's not worry about HP-UX. I've never gotten squeak to run on it, despite considerable trying. And HP-UX isn't ELF (not on 32-bit PA-RISC anyway) so there's no RPATH either. It has its own equivalent way of storing library paths in the SOM binaries. But honestly, I've never heard of any squeakers on HP-UX, so...
- Andrew
vm-dev@lists.squeakfoundation.org
-
Andreas Raab -
Andrew Gaylard -
Bert Freudenberg -
David T. Lewis -
Douglas Brebner -
Eliot Miranda -
Ian Piumarta -
John M McIntosh -
K. K. Subramaniam