MySQL Bugs: #111980: MySQL Server does not read OpenSSL's openssl.cnf file unless ssl-fips-mode=ON

Bug #111980	MySQL Server does not read OpenSSL's openssl.cnf file unless ssl-fips-mode=ON
Submitted:	4 Aug 2023 16:14	Modified:	11 Aug 2023 21:21
Reporter:	Aaron Steigerwald	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: Security: Encryption	Severity:	S3 (Non-critical)
Version:	8.0.34	OS:	Windows (Server 2019 Standard)
Assigned to:		CPU Architecture:	x86
Tags:	openssl

Description:
I compiled MySQL 8.0.34 with OpenSSL 3.0.9 on Windows.

Problem:

The --ssl-fips-mode Server System Variable must be set to ON or STRICT in order for MySQL's OpenSSL capabilities to be configured using the OpenSSL openssl.cnf file. However, "Server System Variables" documentation (https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html) says "As of MySQL 8.0.34, this option [--ssl-fips-mode] is deprecated and made read-only. Expect it to be removed in a future version of MySQL."

In addition, the "FIPS Support" documentation (https://dev.mysql.com/doc/refman/8.0/en/fips-mode.html) says "When MySQL is compiled using OpenSSL 3.0, and an OpenSSL library and FIPS Object Module are available at runtime, the server reads the OpenSSL configuration file and respects the preference to use a FIPS provider, if one is set." The MySQL server does not read the OpenSSL configuration file UNLESS --ssl-fips-mode=ON or STRICT.

In summary, --ssl-fips-mode must be set to ON or STRICT in order for MySQL OpenSSL libraries to read OpenSSL's openssl.cnf configuration file, but --ssl-fips-mode is deprecated and will be removed in a future release.

How to repeat:
1) Use a version of MySQL 8.0.34 that has been compiled with a FIPS-capable OpenSSL 3.0.9.

2) Ensure MySQL is configured to run without the --ssl-fips-mode Server System Variable.

3) Configure OpenSSL to run in FIPS mode. Example openssl.cnf and fipsmodule.cnf config files are provided at the end of this section.

4) Execute "select md5(777);" from a MySQL client connected to the server. It returns a value but it shouldn't because MD5 is not a FIPS-compliant algorithm. It should return all zeros and a warning if MySQL prompted OpenSSL to read the openssl.cnf config file.

5) Set --ssl-fips-mode=ON and restart MySQL.

6) Execute "select md5(777);show warnings;" from a MySQL client connected to the server. It returns all zeros and the following warning: "SSL fips mode error: FIPS mode ON/STRICT: MD5 digest is not supported."

7) Insert a typo in the openssl.cnf, such as "openssl_conf = openssl_initBAD". Restart MySQL. It fails to start because --ssl-fips-mode=ON prompts MySQL's OpenSSL libraries to read the openssl.cnf file, which contains a config error.

8) Remove the --ssl-fips-mode=ON Server System Variable. Restart MySQL. It starts correctly because --ssl-fips-mode is missing and no longer prompts MySQL's OpenSSL libraries to read the openssl.cnf file.

openssl.cnf config:
-----
# This definition stops the following lines choking if HOME isn't
# defined.
HOME = .

# Use this in order to automatically load providers.
openssl_conf = openssl_init

# Comment out the next line to ignore configuration errors
config_diagnostics = 1

# For FIPS
# Optionally include a file that is generated by the OpenSSL fipsinstall
# application. This file contains configuration data required by the OpenSSL
# fips provider. It contains a named section e.g. [fips_sect] which is
# referenced from the [provider_sect] below.
# Refer to the OpenSSL security policy for more information.
.include "C:/Program Files/Common Files/SSL/fipsmodule.cnf"

[openssl_init]
providers = provider_sect
alg_section = algorithm_sect

# List of providers to load
[provider_sect]
default = default_sect
# The fips section name should match the section name inside the
# included fipsmodule.cnf.
fips = fips_sect

# If no providers are activated explicitly, the default one is activated implicitly.
# See man 7 OSSL_PROVIDER-default for more details.
#
# If you add a section explicitly activating any other provider(s), you most
# probably need to explicitly activate the default provider, otherwise it
# becomes unavailable in openssl. As a consequence applications depending on
# OpenSSL may not work correctly which could lead to significant system
# problems including inability to remotely access the system.
[default_sect]
activate = 1

# Ensure all crypto algorithms conform to FIPS standards. An error will be generated if they don't, e.g. MD5 digest.
[algorithm_sect]
default_properties = fips=yes
-----

fipsmodule.cnf config (some values may need to be regenerated, like the MACs):
-----
[fips_sect]
activate = 1
install-version = 1
conditional-errors = 1
security-checks = 1
module-mac = F0:90:F6:FB:3A:43:9F:6B:83:C1:A4:E4:78:C5:EE:0F:90:65:06:DC:E1:36:80:AC:71:3D:47:06:F1:7F:83:C6
install-mac = 41:9C:38:C2:8F:59:09:43:2C:AA:2F:58:36:2D:D9:04:F9:6C:56:8B:09:E0:18:3A:2E:D6:CC:69:05:04:E1:11
install-status = INSTALL_SELF_TEST_KATS_RUN
-----

Suggested fix:
MySQL code must be updated so it prompts its OpenSSL engine to read the OpenSSL configuration file once MySQL detects OpenSSL libraries are available at runtime.

Hi Mr. Aaron,

Thank you very much for your bug report.

However, we do have some additional questions for you .....

Have you tried our binary for Windows and see whether you had problems with it. If you have not done that, please do, because you might have a problem with compiling.

Next, we need a fully repeatable test case. 

That test case should be based on the OpenSSL version  that is required for 8.0.34. We require OpenSSL version 3.3.6. Please, try using that version. That is pre-requisite that MySQL 8.0.34 works correctly with SSL.

The reason why you might have all these problems is that you use too old version of OpenSSL.

Please, re-try with 3.3.6 and let us know your experience.

Hi Mr. Steigerwald,

We have analysed deeper your report and concluded that it is not a bug.

First of all, the OpenSSL library is indeed a tad old. Later MySQL releases require a newer one.

Next, since you have compiled the library yourself, your config file is not where it should be.

Note that the error you are  quoting in the bug is not coming from the openssl APIs, it's the server disabling certain things when --fips-mode is on. If --fips-mode is off, and your library is configured in FIPs mode,  the server would try the MD5 and would most likely  crash or misbehave.

Not a bug.

Hello,

Thank you for taking the time to review my bug submission. I respectively disagree with your determination that it is not a bug. I've addressed your questions and comments below and have included more information.

Have you tried our binary for Windows and see whether you had problems with it. If you have not done that, please do, because you might have a problem with compiling.

Please see the "FIPS Support" section in https://dev.mysql.com/doc/refman/8.0/en/fips-mode.html. It says "MySQL supports FIPS mode, if compiled using OpenSSL 3.0 or OpenSSL 1.0.2, and an OpenSSL library and FIPS Object Module are available at runtime."

Next, we need a fully repeatable test case. 

A fully repeatable test case has been included in the initial bug report.

That test case should be based on the OpenSSL version  that is required for 8.0.34. We require OpenSSL version 3.3.6. Please, try using that version. That is pre-requisite that MySQL 8.0.34 works correctly with SSL.

OpenSSL version 3.3.6 does not exist. The latest version is 3.1.2. The MySQL documentation says OpenSSL version 8.0 is required. Please see the "FIPS Support" section in https://dev.mysql.com/doc/refman/8.0/en/fips-mode.html. It says "MySQL supports FIPS mode, if compiled using OpenSSL 3.0 or OpenSSL 1.0.2, and an OpenSSL library and FIPS Object Module are available at runtime." Additional version and requirements information is specified in the "System Requirements for FIPS Mode in MySQL" section.

First of all, the OpenSSL library is indeed a tad old. Later MySQL releases require a newer one.

I don't think this comment applies to the bug I reported. Nevertheless, OpenSSL 3.0.9 is very current in the 3.0.x version line and is actively supported by the OpenSSL group. 3.0.10 was just released to address security concerns. MySQL 8.0.34, the latest MySQL release in the 8.0.x version line and the version this bug was written against, still supports OpenSSL 3.0.x. I don't think it's unreasonable to expect these versions to be relevant in terms of bug reports. Also, the MySQL source code function I believe contains the bug (see below) is exactly the same in MySQL 8.0.34 and 8.1.0.

Next, since you have compiled the library yourself, your config file is not where it should be.

This is not relevant to the reported bug because the config file location is valid for the OpenSSL version used in our dev, test, and prod environments. Nevertheless, "C:\Program Files\Common Files\SSL" is the default OPENSSLDIR directory used when compiling OpenSSL, which must be done to include FIPS functionality. See the "Installation directories" section in https://github.com/openssl/openssl/blob/master/NOTES-WINDOWS.md for info on the default OPENSSLDIR location. The OPENSSLDIR location gets compiled into the OpenSSL libraries and is used to access the openssl.cnf file. Run the command "openssl version -d" to view the compiled value of OPENSSLDIR. MySQL is not aware of the OPENSSLDIR nor does it need to be. It calls OpenSSL library functions that know the location because it's been compiled into the OpenSSL libraries.

Note that the error you are  quoting in the bug is not coming from the openssl APIs, it's the server disabling certain things when --fips-mode is on. If --fips-mode is off, and your library is configured in FIPs mode,  the server would try the MD5 and would most likely  crash or misbehave.

I assume the error you reference is the "select md5(777);" query I included in the test steps. What you stated is not correct and is at the heart of this bug report. You would see for yourself if you followed my test steps, which include setting --fips-mode=off, configuring OpenSSL for FIPS mode, and executing the "select md5(777);" query that returns a correct MD5 value and does NOT crash or misbehave.

I think the bug exists in the mysql-8.0.34\mysys\my_openssl_fips.cc file's "static int set_fips_mode_inner(int fips_mode)" function, which is the same in MySQL 8.1.0:

/**
  Sets fips mode. On error the error is in the openssl error stack

  @retval 0 failure
  @retval non-0 success
*/
static int set_fips_mode_inner(int fips_mode) {
  int rc = -1;
#if OPENSSL_VERSION_NUMBER >= 0x30000000L
  /* Load FIPS provider when needed. */
  if (fips_mode > 0 && nullptr == ossl_provider_fips) {
    ossl_provider_fips = OSSL_PROVIDER_load(nullptr, "fips");
    if (ossl_provider_fips == nullptr) rc = 0;
  }
  if (rc) rc = EVP_default_properties_enable_fips(nullptr, fips_mode);
#else  /* OPENSSL_VERSION_NUMBER >= 0x30000000L */
  rc = FIPS_mode_set(fips_mode);
#endif /* OPENSSL_VERSION_NUMBER >= 0x30000000L */
  return rc;
}

The OpenSSL FIPS provider is only loaded using the OpenSSL OSSL_PROVIDER_load function when fips_mode > 0, i.e. is set to ON or STRICT. The OpenSSL EVP_default_properties_enable_fips function is called when fips_mode == 0; however, it does not appear to load the openssl.cnf file (which in turn loads the fipsmodule.cnf file) that the OSSL_PROVIDER_load function does and would disable FIPS mode if it did because fips_mode == 0.

See https://www.openssl.org/docs/man3.0/man3/OSSL_PROVIDER_load.html for OSSL_PROVIDER_load details.
See https://www.openssl.org/docs/man3.0/man3/EVP_default_properties_is_fips_enabled.html for EVP_default_properties_enable_fips details.

Hi,

The conclusion that this is not a bug is based on the opinion of the developers who have designed and implemented the feature.

I still believe this is a bug based on the assumption that MySQL 8.0.34 is expected to operate in FIPS mode if:

1) the --ssl-fips-mode server system variable is omitted (default is OFF) or explicitly set to --ssl-fips-mode=OFF.
AND
2) OpenSSL 3.0.X (tested with 3.0.9 and 3.0.10) is configured to operate in FIPS mode.

MySQL 8.0.34 does NOT operate in FIPS mode if the above #'s 1 & 2 are true.

The following is an example of how I verified OpenSSL is configured for FIPS mode at the Windows OS level:
PS C:\> echo '777' | openssl md5
Error setting digest
10060000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto\evp\evp_fetch.c:373:Global default library context, Algorithm (MD5 : 102),Properties ()
10060000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto\evp\digest.c:254:

BTW, the above behavior is not isolated to Windows. I recreated the issue in Linux. As I pointed out in my last message, it will also be a problem in 8.1.0.

I fixed the problem by updating the mysql-8.0.34\mysys\my_openssl_fips.cc file's "static int set_fips_mode_inner(int fips_mode)" function, which is the same in MySQL 8.1.0:

/**
  Sets fips mode. On error the error is in the openssl error stack

  @retval 0 failure
  @retval non-0 success
*/
static int set_fips_mode_inner(int fips_mode) {
  int rc = -1;
#if OPENSSL_VERSION_NUMBER >= 0x30000000L
  if (nullptr == ossl_provider_fips) {
    /* Always try to load FIPS provider if available. It is configured in the openssl.cnf file. */
    if (OSSL_PROVIDER_available(nullptr, "fips")) {
      ossl_provider_fips = OSSL_PROVIDER_load(nullptr, "fips");
    }
  }

  /* Set a failure return code if ossl_provider_fips is still null and fips_mode > 0. */
  if (nullptr == ossl_provider_fips) {
    /* Set rc=1 (success) if fips_mode==0 because it's okay if no FIPS provider is available when FIPS mode is OFF. */
    if (fips_mode == 0) rc = 1; else rc = 0;
  }
  else if (fips_mode > 0) {
    /* Try to ensure FIPS is the default provider, which may override openssl.cnf file settings. */
    rc = EVP_default_properties_enable_fips(nullptr, fips_mode);
  }
#else  /* OPENSSL_VERSION_NUMBER >= 0x30000000L */
  rc = FIPS_mode_set(fips_mode);
#endif /* OPENSSL_VERSION_NUMBER >= 0x30000000L */
  return rc;
}

I compiled and tested it and it works in accordance with the above assumption criteria.

I will not post any more messages. Perhaps my assumption criteria was wrong and this bug report exercise was unnecessary. If I am correct and FIPS mode should work if --ssl-fips-mode is not set or set to OFF, the MySQL development team will discover this was a bug after all when they remove the --ssl-fips-mode server system variable.