Description:
When ran the pstress long-stable pipeline, we found that index page corruption was triggered. The crash stack is as follows:
2024-05-09T02:53:22.384816Z 28 [ERROR] [MY-011853] [InnoDB] Corruption of an index tree: table `pstress`.`tt_15` index `tt_15i2`, father ptr page no 4673, child page no 2134
PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 800746c5; asc F ;;
1: len 4; hex 33654454; asc 3eDT;;
2: len 6; hex 544f41683566; asc TOAh5f;;
3: len 4; hex 80018be7; asc ;;
4: len 4; hex 8005a9dd; asc ;;
5: len 11; hex 6520202020202020202020; asc e ;;
6: len 1; hex 80; asc ;;
7: len 4; hex 33656454; asc 3edT;;
8: len 4; hex 74553348; asc tU3H;;
2024-05-09T02:53:22.385019Z 28 [Note] [MY-012688] [InnoDB] n_owned: 0; heap_no: 2; next rec: 184
PHYSICAL RECORD: n_fields 10; compact format; info bits 0
0: len 4; hex 800739cf; asc 9 ;;
1: len 4; hex 32344665; asc 24Fe;;
2: len 4; hex 656d6f58; asc emoX;;
3: len 4; hex 80095063; asc Pc;;
4: len 4; hex 80033e21; asc >!;;
5: len 11; hex 346c50756e31346a202020; asc 4lPun14j ;;
6: len 1; hex 81; asc ;;
7: len 15; hex 7833656d6f30303631783352783352; asc x3emo0061x3Rx3R;;
8: len 4; hex 714d3934; asc qM94;;
9: len 4; hex 00001241; asc A;;
2024-05-09T02:53:22.385254Z 28 [Note] [MY-012688] [InnoDB] n_owned: 0; heap_no: 29; next rec: 875
2024-05-09T02:53:22.385302Z 28 [ERROR] [MY-011854] [InnoDB] [FATAL] You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html for information about forcing recovery. Then dump + drop + reimport.
2024-05-09T02:53:22.385319Z 28 [ERROR] [MY-013183] [InnoDB] Assertion failure: btr0btr.cc:702:ib::fatal triggered thread 139681740093184
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-05-09T02:53:22Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=3b222c127b490c4851419d93b262a041f58a652e
Thread pointer: 0x7f0a6502a800
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f0a307fd2d0 thread_stack 0x100000
/data/percona/sql/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x5642237cfd7d]
/data/percona/sql/bin/mysqld(print_fatal_signal(int)+0x3b3) [0x564222412333]
/data/percona/sql/bin/mysqld(my_server_abort()+0x66) [0x564222412436]
/data/percona/sql/bin/mysqld(my_abort()+0xa) [0x5642237c884a]
/data/percona/sql/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x323) [0x564223b71763]
/data/percona/sql/bin/mysqld(ib::fatal::~fatal()+0xdf) [0x564223b747cf]
/data/percona/sql/bin/mysqld(+0x2c1175d) [0x564223bb175d]
/data/percona/sql/bin/mysqld(+0x2c11897) [0x564223bb1897]
/data/percona/sql/bin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0x93b) [0x564223bb2dab]
/data/percona/sql/bin/mysqld(btr_cur_pessimistic_update(unsigned long, btr_cur_t*, unsigned long**, mem_block_info_t**, mem_block_info_t*, big_rec_t**, upd_t*, unsigned long, que_thr_t*, unsigned long, unsigned long, mtr_t*, btr_pcur_t*)+0xbd2) [0x564223bc5232]
/data/percona/sql/bin/mysqld(row_ins_sec_index_entry_low(unsigned int, unsigned long, dict_index_t*, mem_block_info_t*, mem_block_info_t*, dtuple_t*, unsigned long, que_thr_t*, bool)+0x835) [0x564223a896e5]
/data/percona/sql/bin/mysqld(row_ins_sec_index_entry(dict_index_t*, dtuple_t*, que_thr_t*, bool)+0x2e0) [0x564223a8f340]
/data/percona/sql/bin/mysqld(+0x2b3d633) [0x564223add633]
/data/percona/sql/bin/mysqld(row_upd_step(que_thr_t*)+0x65d) [0x564223ade04d]
/data/percona/sql/bin/mysqld(+0x2aff206) [0x564223a9f206]
/data/percona/sql/bin/mysqld(ha_innobase::update_row(unsigned char const*, unsigned char*)+0x1c5) [0x5642239214d5]
/data/percona/sql/bin/mysqld(handler::ha_update_row(unsigned char const*, unsigned char*)+0x28b) [0x5642225555bb]
/data/percona/sql/bin/mysqld(Sql_cmd_update::update_single_table(THD*)+0x1cad) [0x56422236d59d]
/data/percona/sql/bin/mysqld(Sql_cmd_dml::execute(THD*)+0x25f) [0x5642222e38af]
/data/percona/sql/bin/mysqld(mysql_execute_command(THD*, bool)+0xce6) [0x564222273f36]
/data/percona/sql/bin/mysqld(dispatch_sql_command(THD*, Parser_state*)+0x440) [0x564222278f50]
/data/percona/sql/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0xf9b) [0x56422227a4cb]
/data/percona/sql/bin/mysqld(do_command(THD*)+0x207) [0x56422227c367]
/data/percona/sql/bin/mysqld(threadpool_process_request(THD*)+0xcf) [0x564224205a8f]
/data/percona/sql/bin/mysqld(+0x3267795) [0x564224207795]
/data/percona/sql/bin/mysqld(+0x31ae895) [0x56422414e895]
/usr/lib64/libpthread.so.0(+0x8f4b) [0x7f0a78b7ef4b]
/usr/lib64/libc.so.6(clone+0x3f) [0x7f0a78666b8f]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f09d3ee4978): UPDATE tt_15 SET c9 = 'd9pftcGFWkDCJFBg' WHERE d10 = 0.00000
Connection ID (thread ID): 28
Status: NOT_KILLED
We can start the mysqld after crash and use check table, it says the index enties miss one.
mysql> check table tt_15;
+---------------+-------+----------+----------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+---------------+-------+----------+----------------------------------------------------------------+
| pstress.tt_15 | check | Warning | InnoDB: Index 'tt_15i2' contains 5675 entries, should be 5674. |
| pstress.tt_15 | check | error | Corrupt |
+---------------+-------+----------+----------------------------------------------------------------+
2 rows in set (0.39 sec)
We can use optimize table to recover the table.
How to repeat:
The table structure is like this. During the test it execute REPLACE、INSERT、SELECT、DELETE、ADD COLUMN、OPTIMIZE、MODIFY COLUMN、DROP COLUMN、UPDATE、ADD INDEX、ADD PARTITION、CHECK TABLE. Don't have any idea to repeat.
CREATE TABLE `tt_15` (
`ipkey` int NOT NULL AUTO_INCREMENT,
`v1` varchar(22) DEFAULT NULL,
`v2` varchar(17) DEFAULT NULL,
`t3` text,
`i4` int DEFAULT NULL,
`c5` char(11) DEFAULT NULL,
`i6` int DEFAULT NULL,
`b7` blob,
`i8` int DEFAULT NULL,
`c9` char(20) DEFAULT NULL,
`d10` double DEFAULT NULL,
`f11` float DEFAULT NULL,
`t12` tinyint(1) DEFAULT NULL,
`g13` varchar(4) GENERATED ALWAYS AS (concat(substr(`i6`,1,1),substr(`c5`,1,1),substr(`c9`,1,1),substr(`v1`,1,1))) VIRTUAL,
`g14` varchar(19) GENERATED ALWAYS AS (concat(substr(`t3`,1,2),substr(`v1`,1,3),substr(`f11`,1,5),substr(`f11`,1,1),substr(`i8`,1,1),`t12`,substr(`t3`,1,3),substr(`t3`,1,3))) VIRTUAL,
`g15` blob GENERATED ALWAYS AS (concat(substr(`i8`,1,1))) VIRTUAL,
PRIMARY KEY (`ipkey`),
KEY `tt_15i0` (`i6`,`ipkey`,`v1` DESC,`c9`,`i4`),
KEY `tt_15i1` (`c5`,`ipkey`),
KEY `tt_15i2` (`ipkey`,`g13`,`v1`,`i8` DESC,`i6`,`c5`,`t12` DESC,`g14`,`b7`(4) DESC)
) ENGINE=InnoDB AUTO_INCREMENT=648831 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=2
