> From: Douglas Brebner <douglas at fang.demon.co.uk>
> 
> > I also hate to think what might happen if ld.so finds a different plugin
> > to the one found subsequently by ioFindExternalFunction().
> 
> On NetBSD dlsym() and friends are handled by ld.so, and I suspect the
> other BSDs are the same, so this problem shouldn't arise on them. I don't
> know how to check the other platforms though.
> 
> Is this really a problem though? I believe dlsym() can only invoke a
> function in the DSO pointed to by the handle returned by dlopen(), so all
> calls to ioFindExternalFunction() should only look in the originally loaded
> plugin

Actually I think it's a non-problem.  When findInternalFunctionIn()
detects the correct entry in the name table it touches the
corresponding field in the address table, which prods ld.so into
looking for the plugin.  ld.so cannot "fail", because dynamic
dependencies are checked at initial load time.  There's no
contradiction there (not even the vaguest possibility of one).

> > It's quite simply *impossible* to link the VM itself correctly unless the
> > .so plugins that it needs (or symbolic links to them, or whatever) are
> > installed in their final location.  This problem affects (at least):
> > SunOS4, AIX, FreeBSD <= 2.2, and (naturally ;-) HP-UX.
> 
> Would using libtool help here?

See below.

> Didn't you get my earlier emails about this?

Nope.  :-(

(If my procmail is doing its job right, it detects Micro$haft/MIME
attachments and pipes them directly into /dev/shredder -- no I'm
joking... well, almost.)

> I also sent a patch for SerialPlugin (one of the speeds isn't
> #ifdef'd but should be)

I already changed it, I think.  (Unless we're talking about a
different speed?  Maybe all speeds > 9600 should be conditional? ;-)

> From: Bert Freudenberg <bert at isg.cs.uni-magdeburg.de>
> 
> > As for the problem of various *nices needing the .so already in place
> > before building the main VM; wow, that's so dumb it's stupid.
> 
> It's so dumb I can't even believe it. So Ian, you're essentially saying
> that it's not possible to dlopen() an arbitrary library that was not
> present at link time?

No, not at all.  Even poor old SunOS knows how to do that (well, with a little
coercion ;-).  I'm saying it's not possible to link the VM at all ("psqueak" or
whatever) unless the linker can find the plugin ".so" files.  See below.

> From: "Lex Spoon" <lex at cc.gatech.edu>
> 
> Perhaps the situation is different on other platforms, of course.

Precisely.

> Regarding the location of .so files, most platforms will let you set an
> environment variable to add to the lookup path.

See above.

> I suggest we do this instead of building a link path into the
> executable on platforms that let you -- the environment variable
> will disappear once the build process is finished, but the built-in
> path will stay in the executable even after its been installed.

The built-in path really is necessary, otherwise ldconfig and/or other
platform dependent stuff would be needed to make the plugins work on
systems where LD_LIBRARY_PATH is not available or is just plain
broken.  And some people claim that ldconfig is pure evil anyway.

What's more, on some systems using -L/.../dir followed by -lFooPlugin
will *hardcode* the location /.../dir/libFooPlugin.so into the
executable, with no recourse to any kind of runtime env var or
whatever.

On some systems you can LD_LIBRARY_PATH, LD_SEARCH_PATH, etc., until
the cows come home and you're *not* going to persuade the linker (note
for anyone who still hasn't realised: we're talking about /bin/ld, NOT
about /lib/ld.so) to look where you want it to.  On these systems you
also seem to be forced to use "-l" to link against the plugins (they
don't understand how to deal with ".so" files on the linker command
line), which means the plugins must be installed in a standard
directory and be called "libXXX.so.X[.Y[.Z]]".

(Aside: I'm still waiting for someone to explain to me why we need to
keep an explicit table of primitive function addresses in the VM at
all [which is the unique cause of all of these problems], when we
could just use the "original" module mechanism to find any "internal"
plugins by gazing into the navel of the executable itself [which it
did already, to find the "intrinsics"].  Or did I miss a good logical
argument against this?)

All this nonsense is kind of a moot point anyway:

> From: Douglas Brebner <douglas at fang.demon.co.uk>
> 
> Would libtool be a better way to handle this, especially for the more
> awkward platforms? Or is that just more complexity for little gain? Ian?

The 2.9a build is entirely automated.  It uses a script to analyse
which files should end up in which plugins/modules/whatever, generates
a corresponding set of "Makefile.am", and uses "aclocal/automake" +
"autoconf" to create the "Makefile.in"s and the configure script.  All
that happens (normally) at "my" end, before distributing the sources.
(On the other hand, updating the "generated" sources from within the
image and rebuilding _might_ require local installations of automake
and autoconf, depending on the presence of new .[ch] files.)  When
building (at "your" end), the configure script builds the required
Makefiles/configured headers, as well as a dedicated libtool
configured for the local platform which is subsequently used to build
the entire kaboodle.

The pain (which was not insignificant ;-) seems to have been well
worth the gain.  Every aspect of platform configuration (choice of
weirdo compile/link flags, how to link against plugins, how and when
to install them and under what names, whether or not to screw dynamic
search paths into executable, etc...) is now *entirely* automated.
"configure.in" is now about 40 lines long, and the templates used to
generate the "Makefile.am"s total no more than a couple of tens of
lines.  The Unix Squeak build process has never been simpler.  (For
those who might already be worrying: _everything_ needed to build
Squeak comes bundled with the sources -- there will be no need to
install any additional tools whatsoever, unless you intend to generate
new plugins or modules from within the image.)

(Caveat: I haven't tried building 2.9 on the more "difficult"
platforms yet, but in principle it should now build perfectly on
several tens of very different Unix platforms -- even the ones nobody
ever tried before.)

The downside is that we have to live with the decisions about naming
and installation that are made by the above tools, and this sometimes
means accepting a "lowest common denominator".  (If it's any
consolation, at least everything will be consistent between the "best"
and "worst" platforms on which Squeak runs -- even if the details
might often seem distastful to those who have never had the chance to
use anything other than the "better" of these systems.)

Another potential source of problems comes from 64-bit systems that
"know" how to compile 32-bit programs.  I am sure that libtool is
capable of doing the right thing (at least on Irix), but it might
require either special options to "configure", or manually setting
"LD[FLAGS]" and/or "CFLAGS" during configuration and/or during the
build: I'll find out when I try building 2.9 on OSF4.  Lunch won't be
free for everybody, I'm afraid...

(But [by far] the best of all: if there are any more portability
problems after this [or if the plugins, or shared libraries for X or
whatever, aren't working "the way they should"] then the argument is
entirely with "configure", and not with me!! ;-)

One final point: the entire suite of configuration and build tools
mentioned above knows (in theory) how to deal with Windows.  Maybe
Andreas and I can figure out how to make it all work nicely both there
and on Unix with a single, common build process.

Ciao,

Ian

PS: I hope Acorns know about all this stuff, otherwise Tim's life
    is about to become "interesting".  (Heh, heh. ;-)

PPS: I hesitated to cc this to the list.  Does anyone beyond the four or
     five people involved in this thread care in the slightest?