MySQL Bugs: #114836: Couldn't load plugin named 'keyring_file' with soname 'keyring

Bug #114836	Couldn't load plugin named 'keyring_file' with soname 'keyring_file.so'
Submitted:	1 May 2024 10:36	Modified:	28 Jun 2024 20:39
Reporter:	Sierk Bornemann	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Installing	Severity:	S2 (Serious)
Version:	MySQL 8.4.0	OS:	MacOS (macOS 14.4.1)
Assigned to:		CPU Architecture:	ARM

Description:
After having upgraded from MySQL 8.3.0 to 8.4.0, the MySQL server doesn't start correctly, starts, stops, hangs in a loop in endless trying, because it doesn't find the deprecated keyring_file plugin anymore, expects a component_keyring_file instead, which isn't also there, isn't provided and changed to by the installer package.
Furthermore, the associated installed launchd file in /Library/LaunchDaemons/com.oracle.oss.mysql.mysqld.plist seems not to be adjusted per default to the new component_keyring_file but still relates to the now deprecated keyring_file plugin and keyring, which doesn't/don't exist there any longer, launchd calls it anyway while starting up MySQL and so causes an error and aborting and shutting down the mysql server starting process through launchd and then retrying it.

mysql-8.4.0-macos14-arm64:
% sudo tail -n30 -f /usr/local/mysql/data/mysqld.local.err

…
2024-04-30T09:02:39.686330Z 0 [System] [MY-015015] [Server] MySQL Server - start.
2024-04-30T09:02:39.831749Z 0 [System] [MY-010116] [Server] /usr/local/mysql/bin/mysqld (mysqld 8.4.0) starting as process 39524
2024-04-30T09:02:39.838725Z 0 [ERROR] [MY-010901] [Server] Can't open shared library '/usr/local/mysql/lib/plugin/keyring_file.so' (errno: 2 dlopen(/usr/local/mysql/lib/plugin/keyring_file.so, 0x0002): tried: '/usr/local/mysql/lib/plugin/keyring_file.so' (no such file)).
2024-04-30T09:02:39.838757Z 0 [ERROR] [MY-010736] [Server] Couldn't load plugin named 'keyring_file' with soname 'keyring_file.so'.
2024-04-30T09:02:39.840453Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-04-30T09:02:39.880879Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-04-30T09:02:39.883407Z 1 [System] [MY-011090] [Server] Data dictionary upgrading from version '80200' to '80300'.
2024-04-30T09:02:40.021018Z 1 [System] [MY-013413] [Server] Data dictionary upgrade from version '80200' to '80300' completed.
2024-04-30T09:02:40.516467Z 4 [System] [MY-013381] [Server] Server upgrade from '80300' to '80400' started.
2024-04-30T09:02:41.662240Z 4 [System] [MY-013381] [Server] Server upgrade from '80300' to '80400' completed.
2024-04-30T09:02:41.738502Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-04-30T09:02:41.738529Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-04-30T09:02:41.741867Z 0 [ERROR] [MY-000067] [Server] unknown variable 'keyring-file-data=/usr/local/mysql/keyring/keyring'.
2024-04-30T09:02:41.742270Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-04-30T09:02:42.727028Z 0 [System] [MY-010910] [Server] /usr/local/mysql/bin/mysqld: Shutdown complete (mysqld 8.4.0)  MySQL Community Server - GPL.
2024-04-30T09:02:42.727051Z 0 [System] [MY-015016] [Server] MySQL Server - end.

---

Default MySQL 8.4.0 /Library/LaunchDaemons/com.oracle.oss.mysql.mysqld.plist on macOS per installer pkg:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>             <string>com.oracle.oss.mysql.mysqld</string>
    <key>ProcessType</key>       <string>Interactive</string>
    <key>Disabled</key>          <false/>
    <key>RunAtLoad</key>         <true/>
    <key>KeepAlive</key>
        <dict>
            <key>SuccessfulExit</key>
            <false/>
            <key>AfterInitialDemand</key>
            <true/>
        </dict>
    <key>SessionCreate</key>     <true/>
    <key>LaunchOnlyOnce</key>    <false/>
    <key>UserName</key>          <string>_mysql</string>
    <key>GroupName</key>         <string>_mysql</string>
    <key>ExitTimeOut</key>       <integer>600</integer>
    <key>Program</key>           <string>/usr/local/mysql/bin/mysqld</string>
    <key>ProgramArguments</key>
        <array>
            <string>/usr/local/mysql/bin/mysqld</string>
            <string>--user=_mysql</string>
            <string>--basedir=/usr/local/mysql</string>
            <string>--datadir=/usr/local/mysql/data</string>
            <string>--plugin-dir=/usr/local/mysql/lib/plugin</string>
            <string>--log-error=/usr/local/mysql/data/mysqld.local.err</string>
            <string>--pid-file=/usr/local/mysql/data/mysqld.local.pid</string>
             <string>--keyring-file-data=/usr/local/mysql/keyring/keyring</string>
             <string>--early-plugin-load=keyring_file=keyring_file.so</string>

        </array>
    <key>WorkingDirectory</key>  <string>/usr/local/mysql</string>
    <key>EnvironmentVariables</key>
        <dict>
            <key>MYSQLD_PARENT_PID</key>
            <string>1</string>
        </dict>
</dict>
</plist>

---

My /Library/LaunchDaemons/com.oracle.oss.mysql.mysqld.plist after the MySQL 8.3.0-to-8.4.0 upgrade:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Disabled</key>
	<false/>
	<key>EnvironmentVariables</key>
	<dict>
		<key>MYSQLD_PARENT_PID</key>
		<string>1</string>
	</dict>
	<key>ExitTimeOut</key>
	<integer>600</integer>
	<key>GroupName</key>
	<string>_mysql</string>
	<key>KeepAlive</key>
	<dict>
		<key>AfterInitialDemand</key>
		<true/>
		<key>SuccessfulExit</key>
		<false/>
	</dict>
	<key>Label</key>
	<string>com.oracle.oss.mysql.mysqld</string>
	<key>LaunchOnlyOnce</key>
	<false/>
	<key>ProcessType</key>
	<string>Interactive</string>
	<key>Program</key>
	<string>/usr/local/mysql/bin/mysqld</string>
	<key>ProgramArguments</key>
	<array>
		<string>/usr/local/mysql/bin/mysqld</string>
		<string>--defaults-file=/usr/local/mysql/my.cnf</string>
		<string>--basedir=/usr/local/mysql</string>
		<string>--datadir=/usr/local/mysql/data</string>
		<string>--plugin-dir=/usr/local/mysql/lib/plugin</string>
		<string>--early-plugin-load=keyring_file=keyring_file.so</string>
		<string>--keyring-file-data=/usr/local/mysql/keyring/keyring</string>
		<string>--log-error=/usr/local/mysql/data/mysqld.local.err</string>
		<string>--pid-file=/usr/local/mysql/data/mysqld.local.pid</string>
		<string>--user=_mysql</string>
	</array>
	<key>RunAtLoad</key>
	<false/>
	<key>SessionCreate</key>
	<true/>
	<key>UserName</key>
	<string>_mysql</string>
	<key>WorkingDirectory</key>
	<string>/usr/local/mysql</string>
</dict>
</plist>

A custom /usr/local/mysql/my.cnf option file does exist, is referenced in the PrefPane, but is an empty file, contains only some default comments.

How to repeat:
Install mysql-8.4.0-macos14-arm64.dmg resp. upgrade from MySQL 8.3.0 to 8.4.0 on macOS. Start MySQL.

See also
https://forums.mysql.com/read.php?11,723964,723964

To get it to start, please disable the "Keyring Data File" option in the preference pane, or remove the two keyring lines from the launchd plist manually.

[quote]

[2 May 11:24] Daniel Fischer

To get it to start, please disable the "Keyring Data File" option in the preference pane, or remove the two keyring lines from the launchd plist manually.

[/quote]

Thanks a lot for the advice. I just did the first (disable the "Keyring Data File" option in the preference pane), the second (remove the two keyring lines from the launchd plist) followed automatically caused by the first. It now works. Thanks.

% sudo tail -n10 -f /usr/local/mysql/data/mysqld.local.err:

2024-05-02T11:34:28.413140Z 0 [System] [MY-015015] [Server] MySQL Server - start.
2024-05-02T11:34:28.561692Z 0 [System] [MY-010116] [Server] /usr/local/mysql/bin/mysqld (mysqld 8.4.0) starting as process 59196
2024-05-02T11:34:28.600928Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-05-02T11:34:28.683336Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-05-02T11:34:28.835154Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-05-02T11:34:28.835184Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-05-02T11:34:28.847256Z 0 [System] [MY-010931] [Server] /usr/local/mysql/bin/mysqld: ready for connections. Version: '8.4.0'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2024-05-02T11:34:29.120782Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /tmp/mysqlx.sock

I also tried to install MySQL version 8.4 LTS the same day it appeared and encountered the reported issue.

Would be really nice if MySQL developers would either fix this issue by updating the installation DMG (as "remove the two keyring lines from the launchd plist") ot at least provide a notice / warning on the download page for macOS users outlining this issue, how to fix etc.

This is not normal that macOS users get to know it post factum.

[quote]
[2 May 15:07] Levan Kekelidze
[/quote]

+1

Hello Sierk Bornemann,

Thank you for the report and feedback.
Workaround for now as  Daniel Fischer commented i.e. "To get it to start, please disable the "Keyring Data File" option in the preference pane, or remove the two keyring lines from the launchd plist manually."

regards,
Umesh

Hi,

thanks for your report,

here is some blogs that might be useful:
 
 https://blogs.oracle.com/mysql/post/component-keyring-encrypted-file
 https://blogs.oracle.com/mysql/post/keyring-components
 https://blogs.oracle.com/mysql/post/keyring-component-to-plugin-migration
 https://blogs.oracle.com/mysql/post/component-keyring-file

So mac users just can't use keyrings now? I think I'm having the same issue. Only when I unchecked the keyring file for configuration, I can start the server. Why we are having this issue?

I used the workaround to allow me to install 8.4 and actually start the server. However that led to the next problem which was that the latest version of MySQL Workbench was unable to create new users in the 8.4 database. It complained about a missing plugin - something to do with the encryption.

So including all the time for backups and restores of all my databases I have finally gone back to 8.1 of the server (Arm64) and 8.032 (Intel) version of Workbench.

I also found that the latest version of workbench seems to have a bug relating to the tabs (or lack of) when you open up multiple sql editors.

MySQL Workbench 8.0 is recommended for MySQL 8.0. Please refer:

https://www.mysql.com/support/supportedplatforms/workbench.html

[quote
[21 May 12:59] Balasubramanian Kandasamy

MySQL Workbench 8.0 is recommended for MySQL 8.0. Please refer:

https://www.mysql.com/support/supportedplatforms/workbench.html
[/quote]

Yes. But:

The Oracle MySQL Blog, July 18, 2023: Introducing MySQL Innovation and Long-Term Support (LTS) versions
https://blogs.oracle.com/mysql/post/introducing-mysql-innovation-and-longterm-support-lts-...

[quote]
…

MySQL Portfolio & Support Lifecycle

LTS Releases will follow the Oracle Lifetime Support Policy (https://www.oracle.com/support/lifetime-support/resources.html), which includes 5 years of premier and 3 years of extended support. Innovation releases will be supported until the next major & minor release.

Most products in MySQL's portfolio will have Innovation and LTS versions:

• MySQL Server, MySQL Shell, MySQL Router, MySQL NDB Cluster will have Innovation and LTS releases.
• MySQL Operator for Kubernetes, with InnoDB ReplicaSet, Cluster and ClusterSet will have Innovation and LTS releases.
• MySQL Connectors will use the latest version used, but will be considered Generally Available and will remain compatible with all supported MySQL Server versions.
• MySQL Workbench will remain on 8.0.x and is Generally Available and will remain compatible with all supported MySQL Server versions.

…
[/quote]

[quote, emphasis added]
• MySQL Workbench will remain on 8.0.x and is Generally Available and _will remain compatible_ with _all_ supported MySQL Server versions.
[/quote, emphasis added]

Seems to be a contradiction to what you correctly point to. Maybe somebody of the MySQL team could bring some light and clarification into that?

Yes - the workaround did let me run mySQL Server 8.4.0 on MacOS - however I found that mySQL Workbench (latest version) failed to let me create a user account with a related error message.

The process to upgrade on Mac seems to be to backup / export the database, uninstall old version, install new version and then restore. Without workbench I find this unnecessarily complex.

Is this a separate bug in Workbench or a related issue with mySQL Server ?

Just switch to the configuration tab on MysQL > Configuration and uncheck the "Key Ring Datafile" and start your Service. This should resolve the issue on 8.4 version.

Error from mysql workbench 8.0.36

Attachment: 2024-06-19_20-58-31.png (image/png, text), 55.07 KiB.

Re my previous post with the image attached. The workaround for 8.4.0 on MacOs may let you start the server but mysql is left broken and unable to create a user. If 8.4.0 is a LTS version should these problems get some attention ?

Posted by developer:
 
Developer feedback provided an explanation and workaround regarding the user creation failure:

User Creation Failure:
---
Workbench sets the default authentication plugin to mysql_native_password, and that plugin is disabled by default as of MySQL Server 8.4.0. This is why Workbench is failing to create the 8.4 user as it's attempting to use a plugin that's not available.

A workaround is to enable mysql_native_password for MySQL Server 8.4, for example the [mysqld] section in your my.cnf could include:

mysql_native_password=ON

That setting was added in MySQL 8.4. Doing so allows creating users in Workbench. Alternatively, edit the Workbench source code (wb_admin_security.py) to have it utilize caching_sha2_password instead of mysql_native_password although I did not confirm this.

In any case, Workbench having a deprecated auth plugin as the default feels like a valid (albeit separate) bug to me. 

Keyring Plugin:
---
As for the keyring plugin issue (this bug), future server releases on macOS will instead install the keyring component thus not require the aforementioned workaround. MySQL Server 8.4.0 removed deprecated keyring plugin functionality in favor of the preferred keyring components e.g., keyring_file was removed in favor of component_keyring_file in MySQL 8.4.0.

Bug Status:
---
Leaving this bug open until related documentation is improved and the auth plugin issue is settled. All feedback is welcome.

Posted by developer:
 
Fixed as of the upcoming MySQL Server 8.4.1 release on macOS, and here's the proposed changelog entry from the documentation team:

---
On macOS, the DMG now installs the keyring component instead 
of installing keyring plugin functionality that was removed 
in MySQL Server 8.4.0.

The 8.4.0 workaround was to disable the "Keyring Data File" 
option in the preference pane or to manually remove the two
keyring lines from the launchd plist.
---

The user creation issues are related to Workbench and the issue was reported to 
Workbench developers, and the Workbench documentation source files now references 
this issue.

Thank you for the bug reports.

Is there a bug report I can monitor relating to the Workbench problem ?

Thanks

Dave