Bug #107914 corruption in the InnoDB tablespace
Submitted: 19 Jul 2022 15:14 Modified: 20 Jul 2022 12:06
Reporter: Jaime Zac Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version: OS:Any
Assigned to: CPU Architecture:Any

[19 Jul 2022 15:14] Jaime Zac
Description:
Hello,

I have a DB in AWS RDS for years and today, suddenly, it crashed as reported bellow. I couldn't recover the data base, but I could restore it point in time and apparently not data was lost.

Could you please help me understand what happened and how to fix to it not happen again?
Attached the log file

How to repeat:
I cannot repeat, because I dont know what causes it.
[19 Jul 2022 15:14] Jaime Zac
error log

Attachment: mysql-error-running.log (application/octet-stream, text), 24.36 KiB.

[19 Jul 2022 15:15] Jaime Zac
its critical
[20 Jul 2022 12:06] MySQL Verification Team
Hi Mr. Zac,

Thank you for your bug report.

However, what you report is a not a bug, but more a support question. 

We have read your log file and we do not see there a fully repeatable test case. Hence , ....

Not enough information was provided for us to be able to handle this bug. Please re-read the instructions at http://bugs.mysql.com/how-to-report.php 

If you can provide a fully repeatable test case, feel free to add it to this bug and change the status back to 'Open'.  

Can't repeat.

Thank you for your interest in MySQL.
[21 Sep 2022 9:15] Aleksei K
Seems that we have the same bug. After mysql version upgrade from 8.0.22 to 8.0.30(24 hours all was fine). We decided to change schema and ...
------
022-09-20T09:23:17.490167Z 27958 [ERROR] [MY-012153] [InnoDB] Trying to access page number 875705389 in space 7312, space name [schema_name_changed]/[table_name_changes], which is outside the tablespace bounds. Byte offset 0, len 16384, i/o type read. If you get this error at mysqld startup, please check that your my.cnf matches the ibdata files that you have in the MySQL server.
2022-09-20T09:23:17.490228Z 27958 [ERROR] [MY-012154] [InnoDB] Server exits.
2022-09-20T09:23:17.491579Z 27958 [ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:7531 thread 22791513839360
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
09:23:17 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x14bb7fa05000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
[21 Sep 2022 9:18] Aleksei K
Sorry, mistype not schema, but table:
ALTER TABLE `y_super_table`    ADD COLUMN `y_super_table_guy` varchar(3) COLLATE utf8mb4_general_ci NOT NULL DEFAULT '2o4' AFTER `super_table_guy`
[21 Sep 2022 12:16] MySQL Verification Team
We inspected the last two asserts .....

Both of these asserts occur very frequently in the situation when a hardware errors occur.

This seems to be your case, unless you use ECC RAM, 2 bits checking 1 bit correcting, and some array of the redundant disks.
[23 Sep 2022 6:05] Aleksei K
we restored database snapshot on other instance and tried to apply similar ALTER statement. 1 try

Attachment: quality-log-problem-mysql-error-running.log.2022-09-22.10.gz (application/x-gzip, text), 3.69 KiB.

[23 Sep 2022 11:41] MySQL Verification Team
Hi Mr. Zac,

This is a forum for the bug reports with repeatable test cases.

Hence, we need a dump of the table in question, so that we can create that table on some MySQL release earlier than 8.0.28 and then to try to do the upgrade to 8.0.30.

The release 8.0.28 is especially important, since both 8.0.28 and 8.0.29 have many bug fixes related to the upgrade process.

If we manage to repeat it, we shall verify this report. Otherwise, it will remain in the same status.

We are waiting on your feedback.
[26 Sep 2022 13:30] Aleksei K
Hi MySQL Verification Team,
We tried to reproduce similar behaviour, but no luck. When we dump and restore same tables with data, error will not occur. For production case we solved it which ANALYZE and OPTIMIZE table. For now all is fine and we continue to monitor it.
Many thanks for your help!
[27 Sep 2022 11:58] MySQL Verification Team
Hi,

In cases like the one that you described, there are two most common causes of the corruption. First one is some upgrade bug, but all of the known ones have been fixed in 8.0.28 and 8.0.29.

Second possibility are transient errors in the hardware. Those errors last for very short time, couple of milliseconds and are, henceforth, impossible to find by any diagnostic software.

That is why serious businesses rely on the parity-checking RAM and redundant arrays of the stripped disks.