Bug #92129 Plugin keyring_file.so API version too different, crashes mysqld on startup

• Submitted: 22 Aug 2018 15:40
• Modified: 12 Sep 2018 19:00
• Reporter: Gary Anderson
• Status: Closed
• Category: MySQL Server: Installing
• Severity: S2 (Serious)
• Version: 5.7.22, 5.7.23
• OS: Red Hat (RHEL 7.5 )
• CPU Architecture: x86 (64-bit)
• Tags: keyring_file, keyring_file_data, keyring_file.so

[22 Aug 2018 15:40] Gary Anderson

Description:
Upgrading MySQL Community version from 5.7.20 to 5.7.23 on a RHEL 7.5 system (64-bit virtual machine on VMware ESXi 6.0).  Upon finishing the upgrade mysqld fails to start.  The following errors are present in the log file:

2018-08-22T14:44:01.183354Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.23) starting as process 2494 ...
2018-08-22T14:44:01.187073Z 0 [ERROR] Can't open shared library 'keyring_file.so' (errno: 0 API version for KEYRING plugin is too di
fferent)
2018-08-22T14:44:01.187094Z 0 [ERROR] Couldn't load plugin named 'keyring_file' with soname 'keyring_file.so'.
...............
2018-08-22T14:44:01.291548Z 0 [ERROR] unknown variable 'keyring_file_data=/var/lib/mysql/keyring'
2018-08-22T14:44:01.291562Z 0 [ERROR] Aborting

Commenting out the variable 'keyring_file_data' in the my.cnf allows the server to start but encrypted table support is missing.

An attempt was made to downgrade to MySQL Community version 5.7.22 from the above upgrade, the same errors and mysqld non-starting issues persisted.

Downgrading to MySQL Community version 5.7.20 (the original version we started with) presented a fully operational mysqld with no errors.

How to repeat:
1. Install MySQL Community version 5.7.20.
2. Configure support for keyring_file plugin per https://dev.mysql.com/doc/refman/5.7/en/keyring-file-plugin.html
3. Upgrade to MySQL Community version 5.7.22 or 5.7.23.
4. Attempt to start mysqld, note it will not start and the log file will show errors as described.

Suggested fix:
Recompile 'keyring_file.so' plugin with the corrected version symbols.

[7 Sep 2018 13:14] Gary Anderson

We have had various results on different existing MySQL installations.  It seems there is one common thread: if the server was previously a MySQL 5.6 server and was upgraded at some point in the past to version 5.7, we see the issue described in the ticket, or similar.

Almost all the servers we have tested with the above upgraded condition have the wrong version of plugins for the currently installed version of MySQL 5.7.20.

One server did not have any plugins in the filesystem at all; we had not used any of them on that installation though, so this fact was missed.  This was surprising because having downloaded and exploded every "mysql-community-server" RPM from 5.7.20 to 5.7.23 they contain plugins in every version, and all have been updated (date, size) on every release.

This issue probably points to a problem with the RPM itself not writing files to the plugins directory.  This may be an issue with the way the permissions were placed on the 5.6.x version of that directory, or something in the RPM install procedure that is skipping the directory for some reason.  However there is no error or notice during installation of the RPM showing this condition, even in the verify step.

[12 Sep 2018 19:00] Gary Anderson

It seems the problem was our fault. In installing MySQL 5.6 on these systems long ago, we had to create our own SystemD startup script for the mysqld as one was not in the package.  This script included a "--plugin-dir=" option in the command line that defined the location of the version 5.6 plugin path.  This path changed in version 5.7, and the SystemD startup script we created for 5.6 was not overwritten on upgrade from 5.6 to 5.7.  This is probably due to the fact that the RPM for 5.7 was not expecting a startup script to have existed already, and no logic is present in the RPM to overwrite an older one (or create an ".rpmnew" file for the new version).  We should have removed the custom startup script before attempting a minor number upgrade.
Closing the ticket, as this is resolved.