| Bug #111106 | MySQL 8.0.33 install fails on Ubuntu 20.04 with FIPS enabled | ||
|---|---|---|---|
| Submitted: | 22 May 2023 15:22 | Modified: | 1 Jun 2023 18:41 |
| Reporter: | Nathan Hoeller | Email Updates: | |
| Status: | Not a Bug | Impact on me: | |
| Category: | MySQL Server | Severity: | S2 (Serious) |
| Version: | 8.0.33 | OS: | Ubuntu (20.04) |
| Assigned to: | CPU Architecture: | x86 | |
| Tags: | FIPS, ubuntu server | ||
[23 May 2023 12:36]
MySQL Verification Team
Hi, Have you tried installing MySQL for Ubuntu from our download site on dev.mysql.com ????/
[23 May 2023 13:57]
Nathan Hoeller
Originally, it installed the upgrade using APT. The upgrade applied automatically through Unattended Upgrades.
I did just run a test using the files directly from the MySQL Downloads page as you suggested, and I'm getting a similar result. When I run this install on the Snapshot before FIPS is enabled, it still installs just fine with no issues. When I install it after FIPS is enabled, it fails with the following status:
```
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2023-05-23 09:25:04 EDT; 26s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 1152 ExecStartPre=/usr/share/mysql-8.0/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 1474 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Main PID: 1474 (code=exited, status=1/FAILURE)
Status: "Server shutdown complete"
Error: 2 (No such file or directory)
```
Here's the output from the MySQL Error Log:
```2023-05-23T13:01:39.266962Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
2023-05-23T13:01:39.267791Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.33) initializing of server in progress as process 3485
2023-05-23T13:01:39.281658Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-23T13:01:40.188227Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-23T13:01:42Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=7e96857c5b3cc354ec3e700b42758fec35af49d9
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x5609638fd241]
/usr/sbin/mysqld(print_fatal_signal(int)+0x393) [0x56096277eae3]
/usr/sbin/mysqld(handle_fatal_signal+0xa5) [0x56096277eb95]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f6313f00420]
/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_CTX_get_security_level+0x4) [0x7f63142df514]
/usr/sbin/mysqld(security_level()+0x23) [0x5609627bf8e3]
/usr/sbin/mysqld(do_auto_cert_generation(ssl_artifacts_status, char const**, char const**, char const**)+0x57) [0x5609627c6987]
/usr/sbin/mysqld(Ssl_init_callback_server_main::provision_certs()+0x72) [0x560962703fe2]
/usr/sbin/mysqld(TLS_channel::singleton_init(Ssl_acceptor_context_container**, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, Ssl_init_callback*, bool)+0x265) [0x5609627024b5]
/usr/sbin/mysqld(mysqld_main(int, char**)+0x3329) [0x560962504809]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f63139cc083]
/usr/sbin/mysqld(_start+0x2e) [0x5609624e307e]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
```
[23 May 2023 14:01]
MySQL Verification Team
If you install it without FIPS and then turn on FIPS does it start? can you enable core and upload log file and core file.
[23 May 2023 15:03]
Nathan Hoeller
I enabled FIPS after MySQL successfully installed, and after rebooting the server, MySQL failed to load. I'd be happy to provide the log file and core file, but I'm not sure how. Can you provide steps for enabling core, or point me in the right direction?
[23 May 2023 19:30]
Nathan Hoeller
Alright, I think I was able to get the core and log data. I just uploaded it through the Oracle SFTP with the file name mysql-bug-data-111106.zip. Please let me know if you need anything else at this time. Here are the actions I performed that were captured in the log and core files: * Downloading the mysql APT Config file from the dev.mysql.com website to /tmp/ directory * $ sudo dpkg -i /tmp/mysql-apt-config_0.8.25-1_all.deb * $ sudo apt-get update * $ sudo apt-get install mysql-server * Enabled core-file by adding the following to /etc/mysql/mysql.conf.d/mysqld.cnf: [mysqld] core-file [mysqld_safe] core-file-size = unlimited * Exclude buffer pool from the core files: * mysql -u root -p * mysql> SET GLOBAL innodb_buffer_pool_in_core_file=OFF; * Installed GDB to view core dump: * $ sudo apt-get install gdb * Enabled FIPS: * $ sudo pro enable fips * Checked the status of MySQL and Ubuntu Pro (Both were showing as active, but a restart was required to fully activate FIPS): * $ sudo ua status * $ sudo systemctl status mysql * $ sudo reboot now
[24 May 2023 13:15]
MySQL Verification Team
Hi Mr. Hoeller, Seems that you have not configured your FIPS so that our software can run on your system. Please read all about it on the following page: https://dev.mysql.com/doc/refman/8.0/en/fips-mode.html This bug is closed now.
[25 May 2023 11:59]
MySQL Verification Team
Hi Mr. Hoeller, We want to make sure that a problem is not on our side. Would you be so kind as to send us your OpenSSL configuration, it's version, your global MySQL settings, which means the entire my.cnf file. Also, send us your TLS settings and your error log (with it's verbosity turned to the maximum value), which should show all the warnings and the erros. Waiting on your feedback.
[25 May 2023 18:05]
Nathan Hoeller
I've uploaded the requested files to the Oracle SFTP under the filename "mysql-bug-data-111106_2.zip". This time I increased the log verbosity to the max value (3). All related core and log files as well as OpenSSL and MySQL configuration files are included. I wasn't sure about the TLS settings though. Apache and SSL were not set up because I wanted to keep the complexity of the server to a minimum while running these tests. If more information is needed, please let me know. So from what I'm reading on the link you sent, it does look like it may be the OpenSSL version 1.1.1f that's causing the problem. I didn't realize that MySQL still required the older version of 1.0.2 to operate in FIPS mode. Since Ubuntu 20.04 comes with 1.1.1, it'll be a challenge to make this work. Do you know if it's possible to install OpenSSL 1.0.2 alongside 1.1.1 and use it only for MySQL?
[1 Jun 2023 18:41]
Nathan Hoeller
I think I found out what was causing the package to fail the upgrade. I had FIPS enabled through Ubuntu Pro, but I didn't have FIPS enabled within MySQL. This apparently was not an issue for the versions prior to 8.0.32, but something in the 8.0.33 upgrade wasn't playing nice with this discrepancy. When I enabled FIPS in MySQL version 8.0.32 and then ran the upgrade, it worked! I also ran this test with OpenSSL 1.1.1f and another test with OpenSSL 1.0.2u. This didn't seem to change the outcome, both versions still allowed the upgrade to apply when FIPS was enabled in MySQL, or fail if it was not enabled. In both cases I'm still getting the SSL Library warning: ``` [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented. ``` but I guess that isn't as serious since it's been showing in previous versions as well. But, the issue still remains; if I try to install MySQL on a server with Ubuntu Pro FIPS enabled, it will fail because it tries to install the newest 8.0.33 version. But if I force it to install 8.0.32 first, then apply FIPS within MySQL, and then upgrade to 8.0.33, it works. Is it possible to install MySQL with FIPS already enabled? If not, it seems there's still a bug here to address.
[11 Mar 14:57]
Manjunath B
I'm also facing a similar crash when enabling in Percona Cluster with Mysql 8.0.35 on Ubuntu 20.04 with FIPS enabled. In this case issue is resolved by enabling fips mode at the MySQL level, but how to we resolve the issue in 8.0.35 where the ssl_fips_mode flag in config is depreciated. Appreciate a response as it blocking usage of 8.0.35-27.1.
Description: Installing MySQL Server 8.0.33 on a Ubuntu 20.04 server with FIPS enabled results in an error. This error does not occur when installing the previous MySQL Server version (8.0.32). The error also does not occur when the server is not running with FIPS enabled. This was discovered after the server ran an unattended upgrade for MySQL on May 10, 2023. The partial package install resulted in a failure to start the MySQL Server. The Error in Console Output after running `$ sudo apt install mysql-server`: ------ 2023-05-22T13:46:13.807257Z 0 [ERROR] [MY-010946] [Server] Failed to start mysqld daemon. Check mysqld error log. Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details. invoke-rc.d: initscript mysql, action "start" failed. ● mysql.service - MySQL Community Server Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Mon 2023-05-22 09:46:17 EDT; 44ms ago Process: 2466 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS) Process: 2494 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE) Main PID: 2494 (code=exited, status=1/FAILURE) Status: "Server shutdown complete" dpkg: error processing package mysql-server-8.0 (--configure): installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1 ------ The Error in MySQL Error Log: ------ 2023-05-22T13:46:09.335227Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented. 2023-05-22T13:46:09.336017Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.20.04.2) initializing of server in progress as process 2284 2023-05-22T13:46:09.349038Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2023-05-22T13:46:10.328756Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2023-05-22T13:46:12Z UTC - mysqld got signal 11 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. BuildID[sha1]=a2a809a7f24eea4fd1a805026e8bbcdde5d2cc2e Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x100000 /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x5589a5ab3a41] /usr/sbin/mysqld(print_fatal_signal(int)+0x39b) [0x5589a4930c5b] /usr/sbin/mysqld(handle_fatal_signal+0xa5) [0x5589a4930d15] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f3f47547420] /lib/x86_64-linux-gnu/libssl.so.1.1(SSL_CTX_get_security_level+0x4) [0x7f3f47948514] /usr/sbin/mysqld(security_level()+0x23) [0x5589a4971ce3] /usr/sbin/mysqld(do_auto_cert_generation(ssl_artifacts_status, char const**, char const**, char const**)+0x57) [0x5589a4978d87] /usr/sbin/mysqld(Ssl_init_callback_server_main::provision_certs()+0x72) [0x5589a48b5fd2] /usr/sbin/mysqld(TLS_channel::singleton_init(Ssl_acceptor_context_container**, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, Ssl_init_callback*, bool)+0x265) [0x5589a48b44a5] /usr/sbin/mysqld(mysqld_main(int, char**)+0x3329) [0x5589a46b5b39] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f3f47011083] /usr/sbin/mysqld(_start+0x2e) [0x5589a4694e8e] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. ------ How to repeat: This can be recreated in the simplest form by following these steps: * Install Ubuntu 20.04.6 LTS on VM VirtualBox 7.0 * Run `$ sudo apt update` * Run `$ sudo apt upgrade` * Removed default actions from GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub (These were preventing FIPS from enabling properly). * Run `$ sudo update-grub` to apply changes. * Install Ubuntu Pro with `$ sudo pro attach <UA Token>` (NOTE: a subscription will be needed. They offer a free tier for personal use). * Reboot Server * Run `$ sudo pro enable fips` * Reboot Server again * Run `$ sudo apt install mysql-server` (This is where failure occurs). I don't think this is a factor in this problem, but in case it's relevant, here is what was was in the "GRUB_CMDLINE_LINUX_DEFAULT" value: GRUB_CMDLINE_LINUX_DEFAULT="auto=true preseed/file=/cdrom/preseed.cfg priority=critical quiet splash noprompt noshell automatic-ubiquity debian-installer/locale=en_US keyboard-configuration/layoutcode=us languagechooser/language-name=English localechooser/supported-locales=en_US.UTF-8 countrychooser/shortlist=US --" Suggested fix: It is possible to completely purge MySQL from the server and then manually reinstall version 8.0.32. If this is done, you'll also need to put a hold on those packages so that they don't automatically update. But this is really just a temporary fix. Ideally, we would want to be running the most up-to-date version.