[Vm-dev] Pharo VM crash in bootstrap with Inconsistency detected by ld.so

Mon Mar 12 08:28:36 UTC 2018

Hi Tobias, Esteban and Fabio,

Attempting to bootstrap the image with Pharo 7.0 VMs newer than 30 Jan
are failing with:

+ vm/pharo Pharo7.0-32bit-eabf524.image eval --save "Metacello new
baseline: 'Tonel';repository:
'github://pharo-vcs/tonel:v1.0.5';onWarning: [ :e | Error signal: e
messageText in: e signalerContext ]; load: 'core'"

...

...finished baselinelinear load :
linear load : baseline [BaselineOfTonel]
load : MonticelloTonel-Core-cypress.1
load : MonticelloTonel-FileSystem-cypress.1
Inconsistency detected by ld.so: dl-close.c: 811: _dl_close: Assertion
`map->l_init_called' failed!

VMs prior to this succeed (e.g. the current stable vm70).

The error is happening after the image is saved, so it is actually
possible to continue and successfully bootstrap the image, but
obviously that is not solving the core issue.

Building a debug VM gives more information:

$ gdb ../vmtarget/lib/pharo/5.0-201803091840-21462-Open-a-FileStream-based-on-fd-or-FILE/pharo
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from
../vmtarget/lib/pharo/5.0-201803091840-21462-Open-a-FileStream-based-on-fd-or-FILE/pharo...done.
(gdb) run --headless Pharo7.0-32bit-eabf524.image eval --save
"Metacello new baseline: 'Tonel';repository:
'github://pharo-vcs/tonel:v1.0.5';onWarning: [ :e | Error signal: e
messageText in: e signalerContext ]; load: 'core'"
Starting program:
/home/alistair/pharo7/bootstrap/pharo/vmtarget/lib/pharo/5.0-201803091840-21462-Open-a-FileStream-based-on-fd-or-FILE/pharo
--headless Pharo7.0-32bit-eabf524.image eval --save "Metacello new
baseline: 'Tonel';repository:
'github://pharo-vcs/tonel:v1.0.5';onWarning: [ :e | Error signal: e
messageText in: e signalerContext ]; load: 'core'"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0xf7a7cb40 (LWP 27000)]
Loading (null)
Loading libssl.so
Found libssl.so at libssl.so.1.0.0
Found symbol ASN1_STRING_data in dlhandle_ssl
Found symbol ASN1_STRING_length in RTLD_DEFAULT
Found symbol BIO_free_all in RTLD_DEFAULT
Found symbol BIO_new in RTLD_DEFAULT
Found symbol BIO_s_mem in RTLD_DEFAULT
Found symbol BIO_ctrl_pending in RTLD_DEFAULT
Found symbol BIO_ctrl in RTLD_DEFAULT
Found symbol BIO_write in RTLD_DEFAULT
Found symbol BIO_read in RTLD_DEFAULT
Found symbol ERR_print_errors_fp in RTLD_DEFAULT
Found symbol SSL_CTX_free in RTLD_DEFAULT
Found symbol SSL_CTX_set_cipher_list in RTLD_DEFAULT
Found symbol SSL_CTX_set_default_verify_paths in RTLD_DEFAULT
Found symbol SSL_CTX_ctrl in RTLD_DEFAULT
Found symbol SSL_CTX_use_PrivateKey_file in RTLD_DEFAULT
Found symbol SSL_CTX_use_certificate_file in RTLD_DEFAULT
Found symbol SSL_accept in RTLD_DEFAULT
Found symbol SSL_connect in RTLD_DEFAULT
Found symbol SSL_free in RTLD_DEFAULT
Found symbol SSL_ctrl in RTLD_DEFAULT
Found symbol SSL_get_error in RTLD_DEFAULT
Found symbol SSL_get_peer_certificate in RTLD_DEFAULT
Found symbol SSL_get_verify_result in RTLD_DEFAULT
Found symbol SSL_new in RTLD_DEFAULT
Found symbol SSL_read in RTLD_DEFAULT
Found symbol SSL_set_accept_state in RTLD_DEFAULT
Found symbol SSL_set_bio in RTLD_DEFAULT
Found symbol SSL_set_connect_state in RTLD_DEFAULT
Found symbol SSL_write in RTLD_DEFAULT
Found symbol X509_NAME_get_text_by_NID in RTLD_DEFAULT
Found symbol X509_get_subject_name in RTLD_DEFAULT
Found symbol X509_get_ext_d2i in RTLD_DEFAULT
Found symbol X509_free in RTLD_DEFAULT
Found symbol X509_check_ip_asc in RTLD_DEFAULT
Found symbol X509_check_host in RTLD_DEFAULT
Loading libcrypto.so
Found libcrypto.so at libcrypto.so.1.0.0
Cannot find SSL_CTX_set_options
Found symbol BIO_test_flags in RTLD_DEFAULT
Cannot find ASN1_STRING_get0_data
Found symbol sk_new_null in RTLD_DEFAULT
Found symbol sk_push in RTLD_DEFAULT
Found symbol sk_free in RTLD_DEFAULT
Found symbol sk_value in RTLD_DEFAULT
Found symbol sk_num in RTLD_DEFAULT
Found symbol sk_pop_free in RTLD_DEFAULT
Found symbol SSLv23_method in RTLD_DEFAULT
Found symbol SSL_CTX_new in RTLD_DEFAULT
Cannot find OPENSSL_sk_new_null
Cannot find OPENSSL_sk_push
Cannot find OPENSSL_sk_free
Cannot find OPENSSL_sk_value
Cannot find OPENSSL_sk_num
Cannot find OPENSSL_sk_pop_free
Cannot find TLS_method
Found symbol SSL_library_init in RTLD_DEFAULT
Found symbol SSL_load_error_strings in RTLD_DEFAULT
Cannot find OPENSSL_init_ssl

Fetched -> BaselineOfTonel-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- github://pharo-vcs/tonel:v1.0.5
Loaded -> BaselineOfTonel-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- github://pharo-vcs/tonel:v1.0.5
Loading baseline of BaselineOfTonel...
Fetched -> MonticelloTonel-Core-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- github://pharo-vcs/tonel:v1.0.5
Fetched -> MonticelloTonel-FileSystem-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- github://pharo-vcs/tonel:v1.0.5
Loaded -> MonticelloTonel-Core-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- cache
Loaded -> MonticelloTonel-FileSystem-cypress.1 ---
github://pharo-vcs/tonel:v1.0.5 --- cache
...finished baselinelinear load :
linear load : baseline [BaselineOfTonel]
load : MonticelloTonel-Core-cypress.1
load : MonticelloTonel-FileSystem-cypress.1
Inconsistency detected by ld.so: dl-close.c: 811: _dl_close: Assertion
`map->l_init_called' failed!
[Thread 0xf7a7cb40 (LWP 27000) exited]
[Inferior 1 (process 26996) exited with code 0177]
(gdb)

The problem occurrs some time between:

VM 30 Jan 2018 14:55 commit 0d66d2a  (bootstrap succeeds)

and:

VM 31 Jan 2018 14:13 commit 0456605  (bootstrap fails)

These are two consecutive VMs on files.pharo.org, so I can't check
commits in between by downloading, and my attempts to build the VMs
failed.

The commit messages are:

commit 045660598ed0a743d9430b600f3b2ac9cf31d4c4
Merge: b251d57 74e443c
Author: Fabio Niephaus <code at fniephaus.com>
Date:   Wed Jan 31 15:13:15 2018 +0100

    Merge pull request #205 from OpenSmalltalk/dl-openssl

    [SqueakSSL] Overlay OpenSSL for linux/unix

commit 74e443c1e2c1492ee7677471730182393f5ae996
Author: Fabio Niephaus <code at fniephaus.com>
Date:   Wed Jan 31 14:35:45 2018 +0100

    Nit picks [ci skip]

commit 438311cb201cfc01dcb03ddcf034cfb5176b57f4
Author: Tobias Pape <tobias at netshed.de>
Date:   Wed Jan 31 14:27:31 2018 +0100

    Refactor the overlay, comments where due

commit b251d57fec7da2f19945c0555b3847f1e397cd71
Merge: 0d66d2a 66d5790
Author: Fabio Niephaus <code at fniephaus.com>
Date:   Wed Jan 31 10:50:39 2018 +0100

    Merge pull request #206 from estebanlm/add-pharo-certificates

    new try with certificates

commit 66d579096b9d954ed208b621770a937639f58031
Author: Esteban Lorenzano <estebanlm at gmail.com>
Date:   Wed Jan 31 09:33:11 2018 +0100

    new try

commit 8e48e6f2a94ecac2010550b68c2ceab6cf7cde91
Author: Tobias Pape <tobias at netshed.de>
Date:   Tue Jan 30 21:15:51 2018 +0100

    Allow for possibly undefined X509 constant

commit 49353ed2783531b3c8fcb7a522dc6f578ea5414c
Author: Tobias Pape <tobias at netshed.de>
Date:   Tue Jan 30 20:09:38 2018 +0100

    [SqueakSSL] Overlay OpenSSL for linux/unix

    When we dynamically link against OpenSSL, the bundles are not portable,
    as CentOS and friends use other SO_NAMEs than Debian and friends. Also,
    soft-fallback for later features such as host name verification is hard.

    When we statically link, we might lack behind the OS, the binaries are
    bigger, and the legal situation is less clear.

    So we now support not linking at all but rather lookup all necessary
    functions/symbols at runtime.

    This can be disabled with SQSSL_OPENSSL_LINKED which effectively results
    in the dynamically-linked behavior. (This is preferable for platform
    builds, eg, debs and rpms)

I'm struggling to figure this one out at the moment and am hoping that
one of you might recognise something to help track it down...

P.S. I also tried compiling with -DSQSSL_OPENSSL_LINKED, but that
produced compiler errors.

Help :-)
Alistair