Bug #97220 mysql-connector-python segfaults
Submitted: 14 Oct 2019 21:25 Modified: 19 Oct 2019 2:59
Reporter: A B Email Updates:
Status: Verified Impact on me:
None 
Category:Connector / Python Severity:S1 (Critical)
Version:8.0.18 OS:Ubuntu (18.04)
Assigned to: CPU Architecture:x86 (x86_64)

[14 Oct 2019 21:25] A B
Description:
mysql.connector.connect(**dbparams) crashes (segmentation fault) if the standard library module "random" is imported before mysql.connector.

Details:

The code inserted below (under "How to repeat") results in a segmentation fault.
If "import random" is commented out, then it works fine.

I am using Python 3.7.3 on Ubuntu 18.04, and a virtual environment with these modules installed:

> pip list
Package                Version
---------------------- -------
mysql-connector-python 8.0.18 
pip                    19.3   
protobuf               3.10.0 
setuptools             41.4.0 
six                    1.12.0 
wheel                  0.33.6

And I am connecting to the MySQL server that ships with Ubuntu
(5.7.27-0ubuntu0.18.04.1), in case this matters.

I hope you can fix this, and thanks for making mysql-connector-python
available!

Sincerely,
Anders Buch

How to repeat:
import random
import mysql.connector
dbparams = dict(host='localhost', user='asbuch', password='****')
mysql.connector.connect(**dbparams)
[16 Oct 2019 12:23] MySQL Verification Team
Hello!

Thank you for the report and feedback.

regards,
Umesh
[17 Oct 2019 15:49] Johannes Schl├╝ter
Posted by developer:
 
Analysis:

The Python `random` module depends on `hash`, which depends on `_hash` which depends on `libyrypt.so`. Loading `random` will therefore load that system library from system path.

The Connector depends on `libssl.so`, which depends on `libcrypto.so` so when loading that the system's runtime linker will see "oh, libcrypto is already loaded, fine" unfortunately it loads our bundled libssl, which doesn't match the system libcrypto.

If the loading order in the script is changed to load random after the connector, the connector will load the bundled libssl, which loads the bundled libcrypto and the random module is happy.

If a user loads another module, which depends on libssl - say `urllib` - then that module will load the system's libssl and the system's libcrypto and the connector will be happy using those.

One might think that an easy fix would be to use the system's openssl and be fine, but that is no solution, since we want to have portable modules working on systems with older libraries. Since all of the loading is done by the runtime linker we only have limited control.

Work-arounds a use can do:

- reorder the `import` statements
- additionally load a module like `urllib`pulling in libssl and matching libcrypto
- enforcing to load libssl by setting `LD_PRELOAD=..../site-packages/mysql-vendor/libcrypto.so.1.1` in environment to force loading of that library and matching libcrypto on Python start

We need further research to find a solution.
[19 Oct 2019 2:59] A B
Thanks for the information.  However, the following code still results in a segmentation fault:

import urllib
import random
import mysql.connector
dbparams = dict(host='localhost', user='asbuch', password='****')
mysql.connector.connect(**dbparams)

The two other methods (reordering imports and setting LD_PRELOAD) work, but I would prefer to be able to import all standard library modules before vendor modules.  I think this is the recommended approach.

Anders
[16 Jun 2020 11:12] Yushan ZHANG
Have the same issue, after I move `import MySQL.connector` to the first line the crash disappeared.

$ mysql --version
mysql  Ver 14.14 Distrib 5.7.29, for Linux (x86_64) using  EditLine wrapper

$ which python3
/usr/bin/python3

$ python3 --version
Python 3.6.9

$ pip3 freeze | grep mysql
mysql==0.0.2
mysql-connector-python==8.0.19
mysqlclient==1.4.6
[16 Jun 2020 22:46] Josh Arenberg
I'm also seeing this issue, but strangely only see it with 8.0.20 and when I downgrade to 8.0.19. Also, even with 8.0.20, I'm only seeing it on my prod container builds and not in dev. However, in the environment where it is impacting, it impacts every time.
[8 Aug 2020 7:09] Christopher Langton
still happening Python 3.8.4 and the only import is "mysql.connector", nothing else!

reproduce using docker for consistency;

FROM python:3.8-slim-buster

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        build-essential \
        zlib1g-dev \
        libssl-dev \
        wget \
        unzip \
        ldnsutils \
        logrotate && \
    apt-get autoremove -y && \
    apt-get clean && \
    rm -rf /tmp/* /var/lib/apt/lists/*

WORKDIR /srv/app

RUN pip install -q --no-cache-dir --isolated -U pip setuptools wheel

ENTRYPOINT ["/usr/bin/env"]
CMD ["/usr/local/bin/python"]

then using a simple main.py

print('print import')
import mysql.connector
print('ok')

produced;

print import
Segmentation fault (core dumped)
[8 Aug 2020 7:15] Christopher Langton
including when using LD_PRELOAD

LD_PRELOAD=/usr/local/lib/python3.8/site-packages/mysql-vendor/libcrypto.so.1.1 python main.py
[9 Jun 2022 16:54] Laszlo Kiss Kollar
I was about to open a new issue for this, but what I want to propose would fix this problem as well: I think the Python wheels should stop manually copying shared libraries into the package, and use auditwheel for this instead.

In mysql-connector-python vendored shared libraries are copied under the extension manually, like "mysql/vendor/private/libkrb5.so.3". This can cause issues if multiple extensions loaded into the same interpreter rely on the same SONAME, as the other extension might pick this version up, which can be ABI-incompatible with it. I am fairly certain that this is the root cause for the segfault in this issue.

The PyPA's recommended solution to avoid such problems is to use the auditwheel tool before uploading the wheel to PyPI, which performs the vendoring in a safe way: it makes shared library dependencies unique to the extension. It does this by making both the file name and the SONAME unique, and updating all DT_NEEDED references in the extension accordingly. For mysql-connector-python this would mean that nothing else will pick up it's bundled shared library dependencies. auditwheel also takes care of the vendoring itself, so that can be dropped from the build process. The PyPA also provides the manylinux containers which contain auditwheel and all other necessary tools to correctly build wheels for many Python versions and platforms.

auditwheel: https://github.com/pypa/auditwheel
manylinux: https://github.com/pypa/manylinux