MySQL Bugs: #111812: Unable to open '#innodb_redo/#ib_redo31

Bug #111812	Unable to open '#innodb_redo/#ib_redo31_tmp', no permission
Submitted:	19 Jul 2023 10:13	Modified:	19 Jul 2023 12:47
Reporter:	zetang zeng (OCA)	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S3 (Non-critical)
Version:	8.0.33	OS:	Linux
Assigned to:	MySQL Verification Team	CPU Architecture:	ARM

Description:
After deployed a three nodes mysql innodb cluster using mysql-8.0.33-linux-glibc2.17-aarch64.tar.gz on kylin v10, we found mysql crashed just one time.

error log:
```
2023-07-13T20:50:30.025659Z 0 [ERROR] [MY-012592] [InnoDB] Operating system error number 13 in a file operation.
2023-07-13T20:50:30.025711Z 0 [ERROR] [MY-012595] [InnoDB] The error means mysqld does not have the access rights to the directory.
2023-07-13T20:50:30.025724Z 0 [ERROR] [MY-012894] [InnoDB] Unable to open '/data00/mysql/data3406/#innodb_redo/#ib_redo31_tmp' (error: 11).
2023-07-13T20:50:30.025741Z 0 [ERROR] [MY-012891] [InnoDB] Failed to create redo log file /data00/mysql/data3406/#innodb_redo/#ib_redo31 (error: 65) for start LSN 1950296064
2023-07-13T20:50:30.025764Z 0 [ERROR] [MY-013864] [InnoDB] [FATAL] Error 65 encountered when writing to the redo log file: /data00/mysql/data3406/#innodb_redo/#ib_redo30.
```

permissions of files (only #ib_redo31_tmp belongs to wrong user):

```
[minibase@aarch64-kylinV10-1 #innodb_redo]$ ll
总用量 1.9G
drwxr-x---  2 minibase minibase 4.0K  7月 19 17:34  .
drwxr-xr-x 15 minibase minibase 4.0K  7月 19 10:26  ..
-rw-r-----  1 minibase minibase  60M  7月 19 06:50 '#ib_redo0'
-rw-r-----  1 minibase minibase  60M  7月 19 17:17 '#ib_redo1'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo10_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo11_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo12_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo13_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo14_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo15_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo16_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo17_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo18_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo19_tmp'
-rw-r-----  1 minibase minibase  60M  7月 19 17:18 '#ib_redo2'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo20_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo21_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo22_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo23_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo24_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo25_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo26_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo27_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo28_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo29_tmp'
-rw-r-----  1 minibase minibase  60M  7月 19 17:42 '#ib_redo3'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo30_tmp'
-rw-r-----  1 root     minibase  60M  7月 18 21:09 '#ib_redo31_tmp'
-rw-r-----  1 minibase minibase  60M  7月 19 18:10 '#ib_redo4'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo5_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo6_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo7_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo8_tmp'
-rw-r-----  1 minibase minibase  60M  7月 18 21:09 '#ib_redo9_tmp'
```

How to repeat:
We deployed on kylin v10 twice with mysql-8.0.33-linux-glibc2.17-aarch64.tar.gz, and it happened twice. We are going to do more test.

Hi Mr. zeng,

Thank you for your bug report.

However, the error that you get means that ownership of some of directories of files on one of those three systems have not been set properly for the user under whose ID mysql server is running.

Please, try to find out the problem. Let us know whether you succeeded.

If you fix that problem and still have troubles, please provide us with all the info so that we may try to repeat the problem.

Actually, the message that you get already indicates what is the problem.

This could be also a problem with two program accessing the same file.

Hi,

Have you noticed that a file with a problem is owned by user `root` ????

That is the wrong ownership.

I noticed, but no one ever to change the owner of `#ib_redo31_tmp`. Mysql just create it

```
[minibase@aarch64-kylinV10-1 #innodb_redo]$stat \#ib_redo31_tmp
  文件：“#ib_redo31_tmp”
  大小：62914560  	块：122880     IO 块：4096   普通文件
设备：10303h/66307d	Inode：344981597   硬链接：1
权限：(0640/-rw-r-----)  Uid：(    0/    root)   Gid：( 1019/minibase)
最近访问：2023-07-18 21:09:08.572227766 +0800
最近更改：2023-07-18 21:09:08.572227766 +0800
最近改动：2023-07-18 21:09:08.572227766 +0800
创建时间：-
```

Hi,

This can happen to a number of reasons. Most often, MySQL server was started with wrong user ID.

If you think that this is a bug, then please provide us with a fully repeatable test case.

If mysql starts with wrong userId, why other redo file has right permission?

Hi,

There are many possible explanations, one of which is that other files have not been created and / or written  in that run.

In any case, we require a fully repeatable test case.

Can't repeat.