[Vm-dev] building SqueakSSL on linux64x64

Fri Jan 21 20:59:57 UTC 2022

Hai!

There's a multitude of answer I could give.
Let's see and try to keep it fun, the C way 
xD

(TL;DR at the bottom around =-~-=)

> On 21. Jan 2022, at 18:25, Eliot Miranda <eliot.miranda at gmail.com> wrote:
> 
> Hi Tobias, Hi All,
> 
>     I'm trying to build a virtend server VM for linux64x64 that includes SqueakSSL as an internal plugin.

There's fun there :D.

>  Building yields a VM but there is no opensll shared object linked in (ldd shows nothing)

Thats expected, see below.

> .  Is one supposed to be unable to build SqueakSSL as an internal plugin?

This has nothing to do with that.

>  There is no mention at all of SqueakSSL in building/linux64x64/HowToBuild.  There should be *some* mention of SqueakSSL in the HowToBuild files.

That's only a half-truth.
That file says:
"""
To find the full set of options via

        ../../platforms/unix/config/configure --help
"""

And incidentally, there is a hint:

" --disable-dynamicopenssl
                          Disable dynamic lookup of OpenSSL, rather link

"

>  Where is doc on how to build the plugin on 64-bit linux?

It is built by default. No need to do stuff in the first place.
Have you watched the configure output?
It says:
"
checking openssl/ssl.h usability... yes
checking openssl/ssl.h presence... yes
checking for openssl/ssl.h... yes
checking for SSL support... OpenSSL (dynamic)
...
creating SqueakSSL/Makefile
"

========

But what does that all mean? How does these bits fit together? Why can this stuff make you unhappy?

Spoiler: the answer to the last question is: OpenSSL, yay :( !
Let's quote the code (Does that rhyme?):

https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakSSL/openssl_overlay.h#L11-L26

=================================================================================
* When we dynamically link against OpenSSL, the bundles are not
* portable, as CentOS and friends use other SO_NAMEs than Debian and
* friends. Also, soft-fallback for later features such as host name
* verification is hard.
*
* This overlay supports using OpenSSL from 0.9.8 to at least 1.1 at runtime.
*
* When we statically link, we might lack behind the OS, the binaries
* are bigger, and the legal situation is less clear.
*
* So we now support not linking at all but rather lookup all necessary
* functions/symbols at runtime.
*
* This can be disabled with SQSSL_OPENSSL_LINKED which effectively
* results in the dynamically-linked behavior. (This is preferable for
* platform builds, eg, debs and rpms)
=================================================================================

Bingo!
There we have it: ldd _cannot_ find openssl in SqueakSSL as it is never linked!
But what then?

We dlopen and dlsym like there is no morning.
See:
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakSSL/openssl_overlay.h#L688
This is called repeatedly via
	_sqo_find
in
	SQO_DECL___ / SQO_DECL_IF (https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakSSL/openssl_overlay.h#L851)
when expanded in
	SQO_DECLARATIONS
(There are a whooping 65 of these thing D: )

That's admittedly really a mouthful. All just to avoid being distro- and openssl-version-specific!
Can't we link statically?
NO, because the Crypto Kraken will emerge and throw Dan J Bernstein at you, along with a fistful of angry Debian maintainers, and nobody wants them to be cross with you, or worse, scottish…
(Joking aside: linking a crypto lib statically is a bad idea™, all sorts of CVS ids will fall on your feet and hurt a lot)

Fun fact: Did you know you cannot dlopen "libc" (or the equivalent "libSystem") on macOS anymore?
Why? Because the library files no longer exist anymore. "But Tobias," people say, "how is any executable staring then?"
Why, using the newfangled dylib cache, of course, which holds the contents of all well-known dylibs on macOS _in memory_!
So no need for a libSystem.dylib! (Truth to be told, you can extract it with some dev tools, tho…)
Fun, fun fact: how do you dlopen/dlsym something from the C standard library then? To _my_ knowledege: the C lib is part of every processes address/library space, so one can employ the fact that you can open _your running process_ via dlopen, and, voilá, the C lib at your tips, among other things.
(NB. Thats one of the reasons for this line:
	https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakSSL/openssl_overlay.h#L708
)

Phew, this out of the way, a few questions remain.

=-~-=
1. Is there a problem?
	Please verify that, indeed, SqueakSSL does not work. Not having libssl/libcrypto in ldd is (by default) _by design_ and has nothing to do with internal/external plugins.
	Check `nm <vmbinary> | grep sqo_OPENSSL_init_ssl` and it should say something with a " B " column in the middle, meaning a (function) variable to be filled at runtime.

2. How to avoid the dynamic approach?
	Answer A is Above: add `--disable-dynamicopenssl` to configure
	Answer B is in the Header file: define SQSSL_OPENSSL_LINKED as part of CFLAGS (in fact, that's what the configure flag does)

3. Do I want to disable the dynamic approach?
	Can you make sure that the binary is run on a similar kind of distro with the _same version of OpenSSL_?
	Yes? Then, sure, go ahead, this should work. But this is typically only the case when you build a system package, like a DEP or RPM for a specific distro, because only _then_ you know what SO_NAME convention, lib location, and OpenSSL version there is.

4. Is Tobias serious about all the above?
	No, not at all. The contraption for dynamically loading OpenSSL was inspired by Qt, which ran into a similar bunch of problems, and while I have learned a lot doing it in 2018, like neat C99 features[1] or library path walking[2], Not needing this kind of Apparillo™ would be amazing. 
	Also, OpenSSL is a strange beast. Don't trust it with any kind of Ring, Gollum, Gollum…
=-~-=

That's all, folks :)

Best regards
	-Tobias

PS: Technology Connections is a neat youtube channel, and they are good at humor, in contrast to me.
PPS: I hate gmail forcing HTML mails on everyone :( (not your fault :) )

[1]: https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakSSL/openssl_overlay.h#L474 
	(look at char (*libs[n]) );
[2]: https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/plugins/SqueakSSL/openssl_overlay.h#L510
	(hey, we can find all library dirs of the running executable!)