Description:
Launched 3 docker containers from a same Docker image in which a MySQL 8.0.32 instance had been installed and populated with a few databases and tables. Then used the MySQL Shell to configure an innodb cluster. After dba.configureInstance() had been called with success, calling cluster.addInstance() to join a MySQL instance failed because of timeout waiting for the restart of the instance.
Tooked a look at the log file of the failed instance, found the following relevant error log:
2023-02-04T07:21:44.699098Z 1 [ERROR] [MY-011976] [InnoDB] [FATAL] Error renaming file from: ./#innodb_red
o to: ./#innodb_redo.#clone_save
2023-02-04T07:21:44.699122Z 1 [ERROR] [MY-013183] [InnoDB] Assertion failure: clone0api.cc:85:ib::fatal tr
iggered thread 139743152158464
straced the mysql server, found that the failed directory renaming was due to a failed call to the rename() system call:
[pid 573] rename("./#innodb_redo", "./#innodb_redo.#clone_save") = -1 EXDEV (Invalid cross-device link)
After googling, found the cause. According to https://docs.docker.com/storage/storagedriver/overlayfs-driver/:
Renaming directories: Calling rename(2) for a directory is allowed only when both the source and the destination path are on the top layer. Otherwise, it returns EXDEV error (“cross-device link not permitted”). Your application needs to be designed to handle EXDEV and fall back to a “copy and unlink” strategy.
According to https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt , this problem can be workarounded by enable the "redirect_dir" feature for the overlayfs file system type:
echo Y | sudo tee /sys/module/overlay/parameters/redirect_dir
However, It would be appreciated that the MySQL server itself can perform fallback to "copy and unlink" just as the mv command does, which is container friendly and user friendly.
How to repeat:
Ubuntu 22.04
Kernel 5.15.0-58-generic
Docker version 20.10.12, build 20.10.12-0ubuntu4
MySQL 8.0.32
MySQL Shell 8.0.32
Suggested fix:
Although a workaround exists, it would be appreciated that the MySQL server itself can perform fallback to "copy and unlink" just as the mv command does, which is container friendly and user friendly.
Description: Launched 3 docker containers from a same Docker image in which a MySQL 8.0.32 instance had been installed and populated with a few databases and tables. Then used the MySQL Shell to configure an innodb cluster. After dba.configureInstance() had been called with success, calling cluster.addInstance() to join a MySQL instance failed because of timeout waiting for the restart of the instance. Tooked a look at the log file of the failed instance, found the following relevant error log: 2023-02-04T07:21:44.699098Z 1 [ERROR] [MY-011976] [InnoDB] [FATAL] Error renaming file from: ./#innodb_red o to: ./#innodb_redo.#clone_save 2023-02-04T07:21:44.699122Z 1 [ERROR] [MY-013183] [InnoDB] Assertion failure: clone0api.cc:85:ib::fatal tr iggered thread 139743152158464 straced the mysql server, found that the failed directory renaming was due to a failed call to the rename() system call: [pid 573] rename("./#innodb_redo", "./#innodb_redo.#clone_save") = -1 EXDEV (Invalid cross-device link) After googling, found the cause. According to https://docs.docker.com/storage/storagedriver/overlayfs-driver/: Renaming directories: Calling rename(2) for a directory is allowed only when both the source and the destination path are on the top layer. Otherwise, it returns EXDEV error (“cross-device link not permitted”). Your application needs to be designed to handle EXDEV and fall back to a “copy and unlink” strategy. According to https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt , this problem can be workarounded by enable the "redirect_dir" feature for the overlayfs file system type: echo Y | sudo tee /sys/module/overlay/parameters/redirect_dir However, It would be appreciated that the MySQL server itself can perform fallback to "copy and unlink" just as the mv command does, which is container friendly and user friendly. How to repeat: Ubuntu 22.04 Kernel 5.15.0-58-generic Docker version 20.10.12, build 20.10.12-0ubuntu4 MySQL 8.0.32 MySQL Shell 8.0.32 Suggested fix: Although a workaround exists, it would be appreciated that the MySQL server itself can perform fallback to "copy and unlink" just as the mv command does, which is container friendly and user friendly.