MySQL Bugs: #106382: 5.7.37 drops support for symbolic links in the data directory

Bug #106382	5.7.37 drops support for symbolic links in the data directory
Submitted:	4 Feb 2022 18:07	Modified:	7 Feb 2022 17:40
Reporter:	Curtis OReilly	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	5.7.37	OS:	Ubuntu (18.04.6 LTS)
Assigned to:		CPU Architecture:	Any
Tags:	symbolic link

Description:
After an unattended upgrade of MySQL by Ubuntu, which upgraded the mysql-server package from 5.7.36 to 5.7.37, the server would not start.  Error from startup is:

2022-02-04T17:46:09.929231Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2022-02-04T17:46:09.929249Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2022-02-04T17:46:09.929261Z 0 [ERROR] InnoDB: os_file_readdir_next_file() returned -1 in directory ./, crash recovery may have failed for some .ibd files!
2022-02-04T17:46:09.929330Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2022-02-04T17:46:10.530398Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2022-02-04T17:46:10.530529Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-02-04T17:46:10.530580Z 0 [ERROR] Failed to initialize builtin plugins.
2022-02-04T17:46:10.530616Z 0 [ERROR] Aborting

After lots of troubleshooting (and making sure Apparmor is disabled) it was discovered that moving databases that are symbolically linked out of the data directory fixes it.

This is despite the fact that symbolic links are allowed:

mysql> show variables like "%sym%";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| have_symlink  | YES   |
+---------------+-------+
1 row in set (0.00 sec)

Both the symbolic link and the target directory are owned by MySQL.  This was working before the unattended update.

How to repeat:
Create a symbolic link in the data directory that points to a directory outside the data directory and restart MySQL.

Suggested fix:
Restore symbolic link support in the data directory.

HI Mr. OReilly,

Thank you for your bug report.

We have tried to repeat the behaviour that you are reporting, but we were not successful.

We upgraded from 5.7.36 to 5.7.37 with an entire schema being pointed to another disk and it worked like a charm.

However, in cases like yours, the operation can fail. This is because you have got the error from the operating system that the permissions are all denied. In that case, the upgrade will not work. But, that is the error of the administrator that has attempted the upgrade and not our bug.

We are not able to repeat your error. We are also not prone to make errors in the installations or upgrades.

Thank you for looking into this.  Good news... I found the problem.

One of the symbolic links was pointing to a directory that had extra permissions enabled ("other" read/execute).  Which is strange, since extra permissions do not usually cause a "permission denied" issue.  So, the fix was running this:

chmod o-rx /path/to/database 

Write "other" permissions were not enabled, but I suspect it would also be an issue, in which case the following would be safer:

chmod o-rwx /path/to/database

Something else I might add... I have no way of knowing if it was the upgrade that caused this problem or if the permissions had changed while mysql was running and that any restart could have caused the issue.

Hi,

The important fact is that you have found the problem and that symbolic links are supported properly in 5.7.37.

Do also note that real permissions are not those of the link , but of the target directory / file.