MySQL Bugs: #113029: Error: /lib64/libstdc++.so.6: cannot allocate memory in static TLS block

Bug #113029	Error: /lib64/libstdc++.so.6: cannot allocate memory in static TLS block
Submitted:	9 Nov 2023 16:17	Modified:	14 Nov 2023 11:13
Reporter:	Sebastien FLAESCH	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: C API (client library)	Severity:	S2 (Serious)
Version:	8.2.0	OS:	Red Hat (8.8)
Assigned to:		CPU Architecture:	x86
Tags:	LD_PRELOAD, libstdc++, Thread Local Storage, TLS block

Description:
Hello,

Platform is RHEL 8.8:

$ uname -a
Linux medusa... 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.8 (Ootpa)

With MySQL 8.0.33 or 8.2.0, we get the following error when loading libmysqlclient.so.* lib, after using some third party libs with additional dependencies:

Error: /lib64/libstdc++.so.6: cannot allocate memory in static TLS block

If we do first load the libmysqlclient first and connect to the DB server, then use the other third party libs, the error does not occur.

The issue can be fixed with following workaround, but is it normal?

export LD_PRELOAD=/lib64/libstdc++.so.6

How to repeat:
Difficult without third party libs

Suggested fix:
We should not have to set LD_PREPLOAD

Hi Mr. Flaesch,

Thank you for your bug report.

This is a problem that can happen with some third party libs, which are evidently necessary for your project.

We would also like to know how do you load third party libraries after you have connected to the server.

But, it is important that you found a workaround.

Meanwhile, can you send us your entire linker command, for the case when it is not working ......

We are waiting on your feedback.

Hello and thanks for considering this issue.

Our solution is a programming language (Genero BDL/4GL), using runtime system (DVM) written in C.

Our main runtime binary (DVM) is not directly linked to libmysqlclient.so.*: We use dlopen() to load our "DB modules" as .so libs specific to a given DB client and linked to that DB client .so lib such as libmysqlclient.so.* .

We face the issue when mixing MySQL client lib with SAP HANA client lib and Microsoft SQL Server lib.

On Linux, the compilation and link commands for our "DB module" is for ex:

cc ... -O2 -W -Wall -pedantic -Wmissing-prototypes -Wno-long-long -Werror -fPIC ...  -c -D mys_8_2 -I/opt3/dbs/mys/8.2/include mys.c -o mys_8_2.o
cc -shared -Xlinker --no-undefined  -o dbmmys_8_2.so mys_8_2.o ... -Wl,-rpath='$ORIGIN/../lib' -L/opt3/dbs/mys/8.2/lib -lmysqlclient

Here the ldd -r outputs for the SAP HANA and MS ODBC libs, if it helps:

SAP HANA Client:

[f4gl@medusa 2.18.24]$ ldd -r libodbcHDB.so
    linux-vdso.so.1 (0x00007ffed9d73000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f9345a65000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f9345861000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f93454cc000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f934514a000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9344f32000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9344d12000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f934494d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f934681f000)
[f4gl@medusa 2.18.24]$ ldd -r libsapcrypto.so 
    linux-vdso.so.1 (0x00007fffba0f0000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f56a5258000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f56a5054000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f56a4e4c000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f56a4c2c000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f56a4867000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f56a5c92000)

Microsoft ODBC for SQL Server:

[f4gl@medusa lib64]$ ldd -r libmsodbcsql-18.3.so.2.1 
	linux-vdso.so.1 (0x00007ffca8d78000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fc3ee9a2000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fc3ee79a000)
	libodbcinst.so.2 => /dbs/64bits/snc/18.3.2.1/uxo/lib64/libodbcinst.so.2 (0x00007fc3ee586000)
	libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fc3ee29b000)
	libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fc3ee046000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fc3edcb1000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fc3ed92f000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc3ed717000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc3ed4f7000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fc3ed132000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc3eefba000)
	libltdl.so.7 => /lib64/libltdl.so.7 (0x00007fc3ecf28000)
	libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fc3ecd11000)
	libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fc3ecb0d000)
	libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fc3ec8fc000)
	libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fc3ec6f8000)
	libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007fc3ec20e000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fc3ebff6000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fc3ebdcb000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fc3ebbb3000)
	libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fc3eb92f000)

Note the various TLS / Crypto libs used (since error message mentions "TLS", I suspect some incompatibility)

Seb

Hi Mr. FLAESCH,

First of all, we must inform you that this is a typical case of the problem solving and not a bug report.

It is the joint opinion of both the MySQL Verification Team and  of the Development Team in charge of the MySQL clients and C API, that this problem is not a bug report. This is problem that you have  requires consultative support.

Hence, we are directing you to this page:

https://www.mysql.com/support/

But, since you are having problems, here we can only give you some short directions.

Hence .......

First of all, you need to use the same ssl library version in their app(s) as we do. Otherwise it will not work.

This includes all the 3d party dlopen libraries etc .... It doesn't truly matter if it's dlopen() or dl.so or similar. 

That is because it all goes into the same namespace.

Based on what you have provided us with, that is all that we can advise.

We do not consider this as a bug report, since you would definitely  need to provide all steps to reproduce the building of the entire system where our C API library  is included. That would also include all source code files and libraries that you are using, including  the sources of the libraries that you have. It would also have to include source trees of your 4GL and other software that is used.

We have provided all the advices from both teams, based on the the info that you have provided us with.

Once again, this is a joint opinion of the Bug Verification and Development  teams that your report is not a bug report. It truly belongs to the consultative support.

Hello,

Just some last info:

We realized that the "TLS" acronym in the error message "cannot allocate memory in static TLS block" does not stand for "Transport Layer Security" but for "Thread Local Storage".

So even if we agree that OpenSSL libs used by various third party libs must match in version number, we believe this error is not related to crypto libs.

We found this old RHE bug report that looks similar, and specific to libmysqlclient:

https://bugzilla.redhat.com/show_bug.cgi?id=1871396

It seems to be aarch64 specific, we have x86_64, but it's similar.

See also https://sourceware.org/bugzilla/show_bug.cgi?id=25051

PS: We have an Oracle Partner Network account with basic support (Silver), so I am not sure we get MySQL support through https://www.mysql.com/support/ without additional fee.

Hi Mr. Flaesch,

TLS stands for both of these acronyms. 

On the other hand , since you have a problem with linking our client library, it is far more probable that TLS means Transport Layer Security.

Do note that our client libraries are thread-safe, but they do not use threads by themselves. However, they use Transport Layer Security in many places.

Regarding your support, if you have MOS user ID and password, then you can use it to file a support ticket.

Hi,

No in fact, when looking at this similar case https://bugzilla.redhat.com/show_bug.cgi?id=1871396, which mentions LD_PRELOAD=/usr/lib64/mysql/libmysqlclient.so.21, and the error message "cannot allocate memory in static TLS block", we really think that in this error message, "TLS" is about "Thread Local Storage."

So I searched in that direction and found:
https://www.gnu.org/software/libc/manual/html_node/Dynamic-Linking-Tunables.html

Then I have tried to set:

export GLIBC_TUNABLES="glibc.rtld.optional_static_tls=0x1000"

But it did not help.

I am not sure what I am doing here.

What means "MOS"?

Hi ,

MOS means My Oracle Support.

Regarding your technical queries, this are conclusions reached by the developers who are making and building those libraries.

Correction (sorry):

The "cannot allocate memory in static TLS block" occurs only when mixing SAP HANA client and MySQL 8.2.0 client libs.

With MS ODBC for SQL Server 18.3.2.1 and MySQL 8.2.0, we have no issue, only with MySQL 8.0.x (but here OpenSSL libs mismatch)

Hi,

Mismatch of the versions proves our diagnosis.

OK but our main problem is the error

"cannot allocate memory in static TLS block"

which is related to Thread Local Storage in our opinion.

So if the conclusion of your dev team was about OpenSSL libs mismatch, they should reconsider this TLS block issue and have a look at the links I provided.

Sorry for the confusion.

BTW I think I will contact our SAP support: Maybe it's a problem with the SAP HANA client lib.

Note also that with our solution, we support various DB engines and we use various DB client libs (Oracle, Informix, PostgreSQL, MySQL, SQL Server), we have tests mixing all of them, and we fact this static TLS block issue only when mixing SAP HANA and MySQL clients.

Hi Mr. FLAESCH,

We have provided you with answers on all of your questions, including TLS.

Regarding having problems only with our C API, this could be due to the fact that other vendors use other crypto libraries or have their own.