Bug #101615 | libmysqlclient >= 5.7.30 crash on pthread end | ||
---|---|---|---|
Submitted: | 15 Nov 2020 21:03 | Modified: | 26 Nov 2020 9:37 |
Reporter: | Adrián G. | Email Updates: | |
Status: | Closed | Impact on me: | |
Category: | MySQL Server: C API (client library) | Severity: | S3 (Non-critical) |
Version: | 5.7 | OS: | Ubuntu (18.04 & 20.04 (at least)) |
Assigned to: | CPU Architecture: | x86 | |
Tags: | dlclose pthread segfault |
[15 Nov 2020 21:03]
Adrián G.
[15 Nov 2020 21:13]
Adrián G.
Executable failing test case
Attachment: mysql_dlclose_crash.zip (application/x-zip-compressed, text), 316.33 KiB.
[15 Nov 2020 22:33]
Adrián G.
Note: I ignore the behaviour on MySQL 8 branch, since it has normal incompatibilities of various kinds (compile-time/runtime) for my 32-bit project, which targets older glibc and such. I'm only refering to MySQL 5.7 in all moment.
[16 Nov 2020 13:01]
MySQL Verification Team
Hi Mr. G, Thank you for your bug report. We do not think that what you are trying to do is supported, but we have to check for all information before running to the conclusion. Hence, please provide the information on what hardware are you actually running our client on, is it 32-bit or 64-bit and more importantly, are libraries and other system software on that Linux 32-bit or 64-bit. We are waiting on your feedback.
[17 Nov 2020 12:46]
MySQL Verification Team
Hi, This issue has been seen on some old versions of Linux , with some old software. Hence, besides what we asked for already, we would need to know whether the binary that you have built resolves exclusively with 32-bit libraries. You can use output from `ldd` and `file` to obtain this info. Also, this behaviour has been noted with some very old GCC compilers, like version 4.*, which was a bug in the compiler and not in our software. We do require your feedback.
[17 Nov 2020 19:28]
Adrián G.
Hello MySQL team, thanks for answering. I'm attaching the requested details. - Compiler used (in test program): g++ (SteamRT 4.8.4-1ubuntu15~12.04+steamrt1.2+srt2) 4.8.4 - ldd and file output from test program: ldd mysql_test linux-gate.so.1 => (0xf7f35000) libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf7f1b000) libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf7f00000) libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf7d53000) /lib/ld-linux.so.2 (0xf7f37000) file mysql_test mysql_test: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x1b41ba0a260df53a0ab395730c9146f582fc0024, not stripped Also, the OS where I compiled my project (and the test case) is Ubuntu 12.04, perhaps I should have mentioned it. I'm glad to hear the roots may be due to older GCC bug. I've tried with Clang 4.8 (max. version supported from base project) and same crash happens. Let me know if you need anything else. Thank you. Best regards, Adrián
[17 Nov 2020 19:29]
Adrián G.
Correction from last comment: Clang 3.8 max.*
[17 Nov 2020 21:20]
Adrián G.
I've compiled and executed the test case under a more recent G++ version and using 64-bit libmysqlclient (5.7.32), and an up-to-date 64-bit environment (Ubuntu 20.04). The same issue is happening. g++ --version g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 On the other hand, if I use GCC instead I get the following runtime error: _ZTVN10__cxxabiv120__si_class_type_infoE (So I think using G++ is fine)
[17 Nov 2020 21:21]
Adrián G.
Sorry, the last error using GCC (9.3.0) was: ./mysql_connect.so: undefined symbol: _ZTVN10__cxxabiv120__si_class_type_infoE
[18 Nov 2020 12:47]
MySQL Verification Team
HI Mr. G. There are two more informations that we require from you, for the time being. With different version of compiler, you have got a different error message. Can you try to use g++ for linking your executable or to add -lstdc++ to the linking phase ???? Also, for all linked libraries, can you please let us know whether those are 32-bit or 64-bit ??? Many thanks in advance ......
[20 Nov 2020 12:59]
Adrián G.
Hi again MySQL team, Think I could have generated confusion, judging by your last message. > "With different version of compiler, you have got a different error message. Can you try to use g++ for linking your executable or to add -lstdc++ to the linking phase ????" - I have been using g++ all the time. Also you probably mean to add the -lstdc++ option to gcc instead. But that isn't fixing the missing C++ ABI symbol "_ZTVN10__cxxabiv120__si_class_type_infoE", which causes error on dlopen() (may be a limitation). The gcc compilation lines I tried (currently on 64-bit Ubuntu 20.04 system, and gcc/g++ 9.3.0-...) are: gcc -lstdc++ -m32 -Iinclude -shared ../mysql_connect.cpp libmysqlclient.a -o mysql_connect.so gcc -lstdc++ -m32 ../mysql_test.cpp -ldl -lpthread -o mysql_test (I've attached code and results later below to visualize it better than the uploaded test case) > "Also, for all linked libraries, can you please let us know whether those are 32-bit or 64-bit ???" - Both my test program (along MySQL-wrapping shared library), static libmysqlclient and system libraries are 32-bit if I compile for "full 32-bit" (and 64-bit if I compile for "full 64-bit"), you could notice it partly from my `file` outputs of my last reply (/lib/i386-linux-gnu path references). I'm not doing any 32/64 bit mixings, which causes compile errors for the most part at least. Now, I'll put the test case. First, there is the "main" file which is ment to be an executable not directly linked with libmysqlclient, which does a dlopen() on the MySQL-wrapping shared library: #include <cstdio> #include <dlfcn.h> #include <pthread.h> void* ThreadRun(void* arg) { void* pLib = dlopen("./mysql_init.so", RTLD_LAZY); if (pLib != NULL) { puts("dlopen() succeeded"); typedef void (*connect_fn)(); connect_fn pConnectFn = (connect_fn)dlsym(pLib, "InitClose"); pConnectFn(); dlclose(pLib); } else { puts(dlerror()); } puts("Finished thread"); return NULL; } int main() { pthread_t my_thread; puts("Creating thread..."); pthread_create(&my_thread, NULL, &ThreadRun, NULL); pthread_join(my_thread, NULL); puts("Finished everything OK"); } // END OF MAIN SOURCE FILE Then, the "wrapping" shared library links statically against libmysqlclient.a (naturally, there would be no point in linking against shared libmysqlclient.so when the wrapping library is already shared and dynamically loaded in runtime): #include <mysql.h> #include <cstdio> extern "C" void InitClose() { MYSQL connection; mysql_init(&connection); mysql_close(&connection); mysql_library_end(); puts("MySQL init/close finished"); } // END OF MYSQL-WRAPPING SHARED LIBRAY SOURCE FILE To check behaviour on both 32-bit and 64-bit builds, currently I have a test source structure as: . ├── compile (compilation-launch script) ├── i686 (32-bit context) │ ├── include (32-bit MySQL 5.7.32 include folder) │ └── libmysqlclient.a (32-bit MySQL 5.7.32 static library) ├── mysql_init.cpp ("main" program source) ├── mysql_test.cpp (shared wrapping library source) └── x86_64 (64-bit context) ├── include (64-bit MySQL 5.7.32 include folder) └── libmysqlclient.a (64-bit MySQL 5.7.32 static library) And the compilation script expands to: #!/bin/bash pushd i686 echo "---- BEGIN i686 TEST ----" g++ -m32 -Iinclude -shared ../mysql_init.cpp libmysqlclient.a -o mysql_init.so && \ g++ -m32 ../mysql_test.cpp -ldl -lpthread -o mysql_test && \ ./mysql_test popd pushd x86_64 echo "---- BEGIN x86_64 TEST ----" g++ -Iinclude -shared ../mysql_init.cpp libmysqlclient.a -o mysql_init.so && \ g++ ../mysql_test.cpp -ldl -lpthread -o mysql_test && \ ./mysql_test popd // END OF COMPILATION SCRIPT The current build environment is: - 64-bit Ubuntu 20.04 (not using any special sandbox, chroot etc.) - g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 - (Equivalent gcc version, but I'm not using it) The original error happened on the older 32 Ubuntu 12.04 chroot environment, which is the real target of the project. But I think that is currently unrelevant given the segfault problem also happens on this updated environment (and project doesn't support 64-bit). Next, in both cases, the execution result is the same: ---- BEGIN i686 TEST ---- ~/Source/mysql_dlclose_crash/i686 ~/Source/mysql_dlclose_crash Creating thread... dlopen() succeeded MySQL init/close finished Finished thread ./compile: line 11: 35301 Segmentation fault (`core' dumped) ./mysql_test ~/Source/mysql_dlclose_crash ---- BEGIN x86_64 TEST ---- ~/Source/mysql_dlclose_crash/x86_64 ~/Source/mysql_dlclose_crash Creating thread... dlopen() succeeded MySQL init/close finished Finished thread ./compile: line 18: 35314 Segmentation fault (`core' dumped) ./mysql_test ~/Source/mysql_dlclose_crash Again here's the GDB stack trace (32-bit): (gdb) bt #0 0xf6f9fdaa in ?? () #1 0xf7f8f7cf in __nptl_deallocate_tsd.part.0 () from /lib/i386-linux-gnu/libpthread.so.0 #2 0xf7f9079d in start_thread () from /lib/i386-linux-gnu/libpthread.so.0 #3 0xf7ea18fa in clone () from /lib/i386-linux-gnu/libc.so.6 + 64-bit stack trace: (gdb) bt #0 0x00007ffff70c3cfa in ?? () #1 0x00007ffff7f8e5a1 in __nptl_deallocate_tsd () at pthread_create.c:301 #2 0x00007ffff7f8f62a in __nptl_deallocate_tsd () at pthread_create.c:256 #3 start_thread (arg=<optimized out>) at pthread_create.c:488 #4 0x00007ffff7eb6293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Finally, I think I could solve my issue by moving dlopen()/dlclose() calls outside thread, which works fine and I'm okay with as long as there isn't another solution to do it within thread. But if you need any other information, you can ask me. Best regards, Adrián
[20 Nov 2020 14:26]
Adrián G.
Sorry, I add a correction from my test source structure (switched descriptions): ├── .............. ├── mysql_init.cpp (shared wrapping library source) ├── mysql_test.cpp ("main" program source) ├── .............. Also, remember to read my previous answer. Thank you.
[20 Nov 2020 14:48]
Tor Didriksen
The missing symbol indicates you have a dependency problem. Linking with libmysqslclient.a rather than .so is problematic, since the static library does not have any dependencies built into it. In my own build environment, I have: ldd ./libmysql/libmysqlclient.so linux-vdso.so.1 (0x00007ffd698a4000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe22434d000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fe224347000) libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007fe2242b1000) libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007fe223fd1000) librt.so.1 => /lib64/librt.so.1 (0x00007fe223fc7000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fe223dcd000) libm.so.6 => /lib64/libm.so.6 (0x00007fe223c85000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe223c6b000) libc.so.6 => /lib64/libc.so.6 (0x00007fe223aa5000) /lib64/ld-linux-x86-64.so.2 (0x00007fe224787000) libz.so.1 => /lib64/libz.so.1 (0x00007fe223a8b000) Notice the version dependency on libstdc++ Also, gcc9 has defaults which do not match mysql builds, we use: -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -std=gnu++03 notice the explicit ABI version, and the non-default -std argument.
[20 Nov 2020 15:04]
MySQL Verification Team
Hi Mr. G., Since it turns out that linking with static library is wrong, can you let us know if you were more successful with shared library .........
[20 Nov 2020 16:59]
Adrián G.
Hi, Sadly, the approach linking shared libmysqlclient.so in my dlopen()'ed mysql_init.so wrapper causes the same segfault on pthread end for both my i686 and x86_64 tests, exactly same GDB stack traces as well. Edited compilation script: EX_FLAGS="-g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -std=gnu++03" pushd i686 echo "---- BEGIN i686 TEST ----" g++ $EX_FLAGS -m32 -Iinclude -L. -shared ../mysql_init.cpp -lmysqlclient -o mysql_init.so && \ g++ $EX_FLAGS -m32 ../mysql_test.cpp -ldl -lpthread -o mysql_test && \ LD_LIBRARY_PATH=. ./mysql_test popd pushd x86_64 echo "---- BEGIN x86_64 TEST ----" g++ $EX_FLAGS -Iinclude -L. -shared ../mysql_init.cpp -lmysqlclient -o mysql_init.so && \ g++ $EX_FLAGS ../mysql_test.cpp -ldl -lpthread -o mysql_test && \ LD_LIBRARY_PATH=. ./mysql_test popd # END OF COMPILATION SCRIPT Note: EX_FLAGS are Tor's flags. Tried both with and without them, issue happens regardless. P.S. I've copied the libmysqlclient.so to the folders, but I had to create the "libmysqlclient.so.20" symlink pointing to the former because even when the wrapper is instructed to link against the original name, the later is demanded by the linked libraries output: adrian@vps:~/Source/mysql_dlclose_crash/i686$ LD_LIBRARY_PATH=. ldd mysql_init.so linux-gate.so.1 (0xf7f99000) libmysqlclient.so.20 => ./libmysqlclient.so.20 (0xf7a05000) libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf77fc000) libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf77d9000) libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf77d3000) librt.so.1 => /lib/i386-linux-gnu/librt.so.1 (0xf77c8000) libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xf75ea000) libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xf74e5000) libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xf74c6000) /lib/ld-linux.so.2 (0xf7f9a000) Can the linker be instructed to respect the original name rather than the .so.20 ending one? Though this is not important for the main issue. Thank you.
[20 Nov 2020 18:10]
MySQL Verification Team
Hi, It is possible that you have wrong libstdc++ library version installed !!!! Can you check that ??? Also, this looks less like our bug and more like your like your environment and versioning problems. Because, we have no problems at building 32-bit 5.7 libmysqlclient library.
[23 Nov 2020 13:24]
Adrián G.
Hello again, I don't think any of my libstdc++ libraries are wrong, accounting I tried to reproduce the case in several environments already. Basic libmysqlclient integration, that is, not employing this more complex pthread-dlopen-dlclose flow but normal linkage works perfectly fine for all recent 5.7 subversions. Instead, this flow only fails on versions >= 5.7.30 from official releases. I invite you to download the attached test case (the first uploaded via Files tab) and try yourself to observe crash, running provided `compile` script (you need to download libmysqlclient >= 5.7.30 manually into the folder). Also, I may have understood Tor's flags wrong, I added these to my test case but he probably meant to apply them in MySQL client compilation itself. Not important, though. Now, I decided to download latest MySQL 5.7 Source from mysql-server GitHub, and managed to compile it successfully on my Ubuntu 20.04 environment (which isn't the real target for my project). I used the following order: cmake -DDOWNLOAD_BOOST=1 -DWITH_BOOST=~/Source/mysql_boost -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_LIBEVENT=system -DWITH_RAPID=OFF && cmake --build . I double checked that generated server version at include/mysql_version.h was 5.7.32. Then I copied the produced libmysqlclient.a to my test case, executed it and completed fine (no segfault). Forgetting about the original issue, since it seems that the manually built library works fine, I see the produced libmysqlclient.so has these additional dependencies, not present in the Community Server releases: libssl.so.1.1 => /lib/i386-linux-gnu/libssl.so.1.1 (0xf7b4f000) libcrypto.so.1.1 => /lib/i386-linux-gnu/libcrypto.so.1.1 (0xf74ab000) libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf7488000) libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf737d000) How would I go about clearing these out? Especially linking libssl/libcrypto statically which I assume the downloadable versions do, so I don't have to link against these later. Thank you.
[24 Nov 2020 8:23]
Tor Didriksen
Hello again. Assuming you have installed static openssl libs alongside the shared ones: (I only have 64bit) find /usr -name libcrypto* /usr/lib/syslog-ng/3.25/libcryptofuncs.so /usr/lib/x86_64-linux-gnu/libcrypto.so /usr/lib/x86_64-linux-gnu/libcrypto.a /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 /usr/lib/x86_64-linux-gnu/pkgconfig/libcrypto.pc cmake <path to src> -DWITH_SSL="/usr" cmake . | grep SSL -- OPENSSL_INCLUDE_DIR = /usr/include -- OPENSSL_LIBRARY = /usr/lib/x86_64-linux-gnu/libssl.a -- OPENSSL_MAJOR_VERSION = 1 -- OPENSSL_MINOR_VERSION = 01 -- OPENSSL_FIX_VERSION = 01 -- SSL_LIBRARIES = imported_openssl;imported_crypto;dl -- OpenSSL include: /usr/include -- OpenSSL lib: make libmysql ldd ./libmysql/libmysqlclient.so linux-vdso.so.1 (0x00007ffff1b36000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f91cdfd6000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f91cdfb3000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f91cdbdf000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f91cd9c7000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f91cd7d5000) /lib64/ld-linux-x86-64.so.2 (0x00007f91ce69b000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f91cd684000) You are right, on many platforms we build openssl libraries internally, and link them with mysql binaries. For 5.7 we use static openssl libraries. If you want to know why the above works (at least for me) read cmake/ssl.cmake: for custom-built SSL (look for WITH_SSL_PATH) we will prefer the .a libs over .so libs. On Ubuntu 20, I see: pkg-config --libs --static libcrypto -lcrypto -ldl -pthread so you still need to link with libdl.so and libpthread.so
[24 Nov 2020 15:25]
Adrián G.
Hi Tor, Thank you for the input. I've also managed to compile the client on the older environment I need to target (Ubuntu 12.04, 32 bit), first without static OpenSSL linking, using the commands: cmake -DBUILD_CONFIG=mysql_release -DDOWNLOAD_BOOST=1 -DWITH_BOOST=~/my_boost make But, while the following partial output shows in this case: 2020-11-24T12:53:26.7733645Z -- OPENSSL_INCLUDE_DIR = /usr/include 2020-11-24T12:53:26.7734293Z -- OPENSSL_LIBRARY = /usr/lib/i386-linux-gnu/libssl.so 2020-11-24T12:53:26.7734810Z -- CRYPTO_LIBRARY = /usr/lib/i386-linux-gnu/libcrypto.so 2020-11-24T12:53:26.7735275Z -- OPENSSL_MAJOR_VERSION = 1 2020-11-24T12:53:26.7735709Z -- OPENSSL_MINOR_VERSION = 00 2020-11-24T12:53:26.7736108Z -- OPENSSL_FIX_VERSION = 01 2020-11-24T12:53:26.7746305Z -- Looking for SHA512_DIGEST_LENGTH 2020-11-24T12:53:26.8286308Z -- Looking for SHA512_DIGEST_LENGTH - found [................] If I instead compile with your correct static SSL-enabling line: cmake -DBUILD_CONFIG=mysql_release -DDOWNLOAD_BOOST=1 -DWITH_BOOST=~/my_boost -DWITH_SSL=/usr make An error arises in that section: 2020-11-24T13:03:57.5587721Z -- OPENSSL_INCLUDE_DIR = /usr/include 2020-11-24T13:03:57.5588250Z -- OPENSSL_LIBRARY = /usr/lib/i386-linux-gnu/libssl.a 2020-11-24T13:03:57.5588709Z -- CRYPTO_LIBRARY = /usr/lib/i386-linux-gnu/libcrypto.a 2020-11-24T13:03:57.5589094Z -- OPENSSL_MAJOR_VERSION = 1 2020-11-24T13:03:57.5589460Z -- OPENSSL_MINOR_VERSION = 00 2020-11-24T13:03:57.5589797Z -- OPENSSL_FIX_VERSION = 01 2020-11-24T13:03:57.5590040Z CMake Error at cmake/ssl.cmake:247 (MESSAGE): 2020-11-24T13:03:57.5590300Z SSL version must be at least 1.1.1 2020-11-24T13:03:57.5590532Z Call Stack (most recent call first): 2020-11-24T13:03:57.5591194Z CMakeLists.txt:580 (MYSQL_CHECK_SSL) 2020-11-24T13:03:57.5591531Z 2020-11-24T13:03:57.5591625Z 2020-11-24T13:03:57.5605361Z -- Configuring incomplete, errors occurred! As you can see, both SSL versions reported are exactly same in both compilations. Note the following query on the target environment: $ strings /usr/lib/i386-linux-gnu/libssl.a | grep OpenSSL SSLv3 part of OpenSSL 1.0.1 14 Mar 2012 How to solve this? Thank you.
[24 Nov 2020 15:52]
Tor Didriksen
That version check is there to catch errors in our own builds: we are not allowed to ship software with older SSL versions. You can of course build whatever you want. There's no option to disable the verson check, so I'm afraid you have to edit the cmake source: # Static SSL libraries? we require at least 1.1.1 IF(STATIC_SSL_LIBRARY OR (WIN32 AND WITH_SSL_PATH)) IF(OPENSSL_VERSION VERSION_LESS "1.1.1") RESET_SSL_VARIABLES() MESSAGE(FATAL_ERROR "SSL version must be at least 1.1.1") ENDIF() SET(SSL_DEFINES "-DHAVE_STATIC_OPENSSL") ENDIF() Just substitute whatever you want for the "1.1.1"
[26 Nov 2020 9:20]
Adrián G.
Hi, I've finally self built MySQL 5.7.32 client library with static OpenSSL (1.1.0) targetting the real environment (Ubuntu 12.04) and, after proper testing, my reported issue has been solved that way as well within the real project. apt-get install -y bison libaio-dev cmake -DBUILD_CONFIG=mysql_release -DDOWNLOAD_BOOST=1 -DWITH_BOOST=~/my_boost -DWITH_SSL=~/ssl make While I've solved this issue by this mean, recall that it's still happening under the described conditions. Since the behaviour is fine with my self built library, clearly the MySQL client implementation is fine for the matter. I'm sure there has been some library-wise system/dependencies change/s between the official build environments used for 5.7.29 and 5.7.30. The door is open for you to play around. Nevertheless, you're free to close my issue. Thanks. Have a good day, Adrián
[26 Nov 2020 9:37]
MySQL Verification Team
Thank you for the feedback.