Bug #106382 5.7.37 drops support for symbolic links in the data directory
Submitted: 4 Feb 2022 18:07 Modified: 7 Feb 2022 17:40
Reporter: Curtis OReilly Email Updates:
Status: Can't repeat Impact on me:
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.7.37 OS:Ubuntu (18.04.6 LTS)
Assigned to: CPU Architecture:Any
Tags: symbolic link

[4 Feb 2022 18:07] Curtis OReilly
After an unattended upgrade of MySQL by Ubuntu, which upgraded the mysql-server package from 5.7.36 to 5.7.37, the server would not start.  Error from startup is:

2022-02-04T17:46:09.929231Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2022-02-04T17:46:09.929249Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2022-02-04T17:46:09.929261Z 0 [ERROR] InnoDB: os_file_readdir_next_file() returned -1 in directory ./, crash recovery may have failed for some .ibd files!
2022-02-04T17:46:09.929330Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2022-02-04T17:46:10.530398Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2022-02-04T17:46:10.530529Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-02-04T17:46:10.530580Z 0 [ERROR] Failed to initialize builtin plugins.
2022-02-04T17:46:10.530616Z 0 [ERROR] Aborting

After lots of troubleshooting (and making sure Apparmor is disabled) it was discovered that moving databases that are symbolically linked out of the data directory fixes it.

This is despite the fact that symbolic links are allowed:

mysql> show variables like "%sym%";
| Variable_name | Value |
| have_symlink  | YES   |
1 row in set (0.00 sec)

Both the symbolic link and the target directory are owned by MySQL.  This was working before the unattended update.

How to repeat:
Create a symbolic link in the data directory that points to a directory outside the data directory and restart MySQL.

Suggested fix:
Restore symbolic link support in the data directory.
[7 Feb 2022 13:42] MySQL Verification Team
HI Mr. OReilly,

Thank you for your bug report.

We have tried to repeat the behaviour that you are reporting, but we were not successful.

We upgraded from 5.7.36 to 5.7.37 with an entire schema being pointed to another disk and it worked like a charm.

However, in cases like yours, the operation can fail. This is because you have got the error from the operating system that the permissions are all denied. In that case, the upgrade will not work. But, that is the error of the administrator that has attempted the upgrade and not our bug.

We are not able to repeat your error. We are also not prone to make errors in the installations or upgrades.
[7 Feb 2022 17:37] Curtis OReilly
Thank you for looking into this.  Good news... I found the problem.

One of the symbolic links was pointing to a directory that had extra permissions enabled ("other" read/execute).  Which is strange, since extra permissions do not usually cause a "permission denied" issue.  So, the fix was running this:

chmod o-rx /path/to/database 

Write "other" permissions were not enabled, but I suspect it would also be an issue, in which case the following would be safer:

chmod o-rwx /path/to/database
[7 Feb 2022 17:40] Curtis OReilly
Something else I might add... I have no way of knowing if it was the upgrade that caused this problem or if the permissions had changed while mysql was running and that any restart could have caused the issue.
[8 Feb 2022 13:32] MySQL Verification Team

The important fact is that you have found the problem and that symbolic links are supported properly in 5.7.37.

Do also note that real permissions are not those of the link , but of the target directory / file.