MySQL Bugs: #111610: Apparent corruption of an index page to be written to data file.

Bug #111610	Apparent corruption of an index page to be written to data file.
Submitted:	28 Jun 2023 16:40	Modified:	12 Jul 2023 12:57
Reporter:	Trimurthy Pothanaboyina	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	8.0.32	OS:	CentOS (CentOS Linux 7)
Assigned to:		CPU Architecture:	x86 (Ix86_64 : ntel(R) Xeon(R) CPU @ 2.20GHz)

Description:
After upgrading MySQL from 5.7.27 to 8.0.32 we observed DB instance crashed with the following error messages.

2023-06-28T12:22:53.764914Z 0 [ERROR] [MY-012719] [InnoDB] Free list record 14490 is above rec heap top 8605
 len 16384; hex 422cce6400000003ffffffffffffffff0000004e7a2c8c3745bf0000000000000000000002470008219d803a206b1284000000050000001a0000000000000000000000000000000006f9000002470000000200f200000247000000020032010002001e696e66696d756d0003000b000073757072656d756d0a2424
...
...
...
InnoDB: End of page dump
InnoDB: Page may be an index page where index id is 1785
2023-06-28T12:22:53.909089Z 0 [ERROR] [MY-011937] [InnoDB] [FATAL] Apparent corruption of an index page [page id: space=583, page number=3] to be written to data file. We intentionally crash the server to prevent corrupt data from ending up in data files.
2023-06-28T12:22:53.909205Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: buf0dblwr.cc:1538:ib::fatal triggered thread 140236835182336
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2023-06-28T12:22:53Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=f5a4b8541445e3f1cd10798a1d9f0ca415d73180
Server Version: 8.0.32-24 Percona Server (GPL), Release 24, Revision e5c6e9d2

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x2153fed]
/usr/sbin/mysqld(print_fatal_signal(int)+0x39f) [0x11f020f]
/usr/sbin/mysqld(my_server_abort()+0x7e) [0x11f03be]
/usr/sbin/mysqld(my_abort()+0xa) [0x214dfaa]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x31f) [0x24158cf]
/usr/sbin/mysqld() [0x24182bf]
/usr/sbin/mysqld(Double_write::croak(buf_block_t const*)+0xbb) [0x247cbfb]
/usr/sbin/mysqld(Double_write::prepare(buf_page_t const*, void**, unsigned int*)+0x14d) [0x247d01d]
/usr/sbin/mysqld(Double_write::enqueue(buf_flush_t, buf_page_t*, file::Block const*)+0x8a) [0x2488fba]
/usr/sbin/mysqld(dblwr::write(buf_flush_t, buf_page_t*, bool)+0x1ca) [0x248970a]
/usr/sbin/mysqld(buf_flush_page(buf_pool_t*, buf_page_t*, buf_flush_t, bool)+0x26f) [0x248e9bf]
/usr/sbin/mysqld() [0x24913c3]
/usr/sbin/mysqld(buf_flush_do_batch(buf_pool_t*, buf_flush_t, unsigned long, unsigned long, unsigned long*)+0x3ab) [0x2491b6b]
/usr/sbin/mysqld() [0x249320e]
/usr/sbin/mysqld() [0x24948c5]
/usr/sbin/mysqld(void Detached_thread::operator()<void (*)()>(void (*&&)())+0xc2) [0x2301ec2]
/usr/sbin/mysqld() [0x2b367d4]
/lib64/libpthread.so.0(+0x7ea5) [0x7f8c22384ea5]
/lib64/libc.so.6(clone+0x6d) [0x7f8c2073fb0d]
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

How to repeat:
Unfortunately coudn't repeat it manually.

Another Crash Logs from mysql error log.

2023-06-21T12:40:05.731126Z 386184 [ERROR] [MY-012869] [InnoDB] Record in index `role_name_uq` of table `p3000`.`role` was not found on update: TUPLE (info_bits=0, 3 n_cmp=3, fields): {[37]  $ - DevicesSettingsTestRoleFEIKKPZP(0x000024b22dee4465766963657353657474696e677354657374526f6c654645494b4b505a50),[24]39db4cb6-ee9b-4573-8bc4-a7fd0764c4f8(0x33396462346362362d656539622d343537332d386263342d613766643037363463346638),[24]9025f1ec-741e-4198-a075-e44fa6a00332(0x39303235663165632d373431652d343139382d613037352d653434666136613030333332)} at: COMPACT RECORD(info_bits=0, 1 fields): {[8]infimum (0x696e66696d756d00)}
2023-06-21T12:40:07.097240Z 385071 [ERROR] [MY-012869] [InnoDB] Record in index `role_name_uq` of table `p3000`.`role` was not found on update: TUPLE (info_bits=0, 3 n_cmp=3, fields): {[37]   >/ DevicesSettingsTestRoleIXCLWWWW(0x0000083e2ffb4465766963657353657474696e677354657374526f6c654958434c57575757),[24]39db4cb6-ee9b-4573-8bc4-a7fd0764c4f8(0x33396462346362362d656539622d343537332d386263342d613766643037363463346638),[24]74b4c248-e11e-418f-adde-7b1a1a7a0633(0x37346234633234382d653131652d343138662d616464652d376231613161376130363333)} at: COMPACT RECORD(info_bits=0, 1 fields): {[8]infimum (0x696e66696d756d00)}
2023-06-21T12:40:08.863194Z 388427 [ERROR] [MY-012869] [InnoDB] Record in index `role_name_uq` of table `p3000`.`role` was not found on update: TUPLE (info_bits=0, 3 n_cmp=3, fields): {[37]  '  QDevicesSettingsTestRoleCPNIEHVD(0x000027b701514465766963657353657474696e677354657374526f6c6543504e4945485644),[24]39db4cb6-ee9b-4573-8bc4-a7fd0764c4f8(0x33396462346362362d656539622d343537332d386263342d613766643037363463346638),[24]ed051e62-feff-4ff6-871c-b04b47e43318(0x65643035316536322d666566662d346666362d383731632d623034623437653433333138)} at: COMPACT RECORD(info_bits=0, 1 fields): {[8]infimum (0x696e66696d756d00)}
2023-06-21T12:51:20.751416Z 388430 [ERROR] [MY-012687] [InnoDB] [FATAL] Rec offset 99, cur1 offset 10865, cur2 offset 15950
2023-06-21T12:51:20.751563Z 388430 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.cc:503:ib::fatal triggered thread 140115265873664
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2023-06-21T12:51:20Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=f5a4b8541445e3f1cd10798a1d9f0ca415d73180
Server Version: 8.0.32-24 Percona Server (GPL), Release 24, Revision e5c6e9d2

Thread pointer: 0x7f6f6689c000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f6f20a65af0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x2153fed]
/usr/sbin/mysqld(print_fatal_signal(int)+0x39f) [0x11f020f]
/usr/sbin/mysqld(my_server_abort()+0x7e) [0x11f03be]
/usr/sbin/mysqld(my_abort()+0xa) [0x214dfaa]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x31f) [0x24158cf]
/usr/sbin/mysqld() [0x24182bf]
/usr/sbin/mysqld(page_copy_rec_list_end_no_locks(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mtr_t*)+0x353) [0x233f2c3]
/usr/sbin/mysqld(btr_page_reorganize_low(bool, unsigned long, page_cur_t*, dict_index_t*, mtr_t*)+0x1de) [0x2441fbe]
/usr/sbin/mysqld(btr_cur_optimistic_insert(unsigned long, btr_cur_t*, unsigned long**, mem_block_info_t**, dtuple_t*, unsigned char**, big_rec_t**, que_thr_t*, mtr_t*)+0x6cf) [0x245628f]
/usr/sbin/mysqld(row_ins_clust_index_entry_low(unsigned int, unsigned long, dict_index_t*, unsigned long, dtuple_t*, que_thr_t*, bool)+0x460) [0x2363340]
/usr/sbin/mysqld(row_ins_clust_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool)+0xa2) [0x23697b2]
/usr/sbin/mysqld(row_ins_step(que_thr_t*)+0x480) [0x236a250]
/usr/sbin/mysqld() [0x2378e0c]
/usr/sbin/mysqld(ha_innobase::write_row(unsigned char*)+0x339) [0x2258fb9]
/usr/sbin/mysqld(handler::ha_write_row(unsigned char*)+0x1f8) [0xd6f158]
/usr/sbin/mysqld(write_record(THD*, TABLE*, COPY_INFO*, COPY_INFO*)+0x634) [0x10311f4]
/usr/sbin/mysqld(Sql_cmd_insert_values::execute_inner(THD*)+0xa09) [0x1032d79]
/usr/sbin/mysqld(Sql_cmd_dml::execute(THD*)+0x189) [0x10c86b9]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0xb46) [0x106a9b6]
/usr/sbin/mysqld(dispatch_sql_command(THD*, Parser_state*, bool)+0x5c0) [0x106e9e0]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x21b4) [0x1071154]
/usr/sbin/mysqld(do_command(THD*)+0x282) [0x1072032]
/usr/sbin/mysqld() [0x11e01a0]
/usr/sbin/mysqld() [0x2620705]
/lib64/libpthread.so.0(+0x7ea5) [0x7f72689cfea5]
/lib64/libc.so.6(clone+0x6d) [0x7f7266d8c96d]

Trying to get some variables.

Crashed again while accessing the same table

Attachment: crash-log.txt (text/plain), 12.60 KiB.

Crashed again while accessing the same table

Attachment: crash-log.txt (text/plain), 10.69 KiB.

HI Mr. Pothanaboyina,

Thank you for your bug report. However .......

Not enough information was provided for us to be able to handle this bug. Please re-read the instructions at http://bugs.mysql.com/how-to-report.php. 

We have had several bug reports with the same stacktrace, but none of the reporters was able to provide a test case. We can not move forward without a test case.

If you can provide more information and we are able to repeat the corruption, we shall run the test case and change the status of this report.

Thank you for your interest in MySQL.

Can't repeat.

Upgrading to MySQL 8.0.33 resolved the issue for us.

Thank you for your feedback.

This report is closed now.