| Bug #117135 | When executing online ddl rebuilt table, concurrent delete operations may cause row_log memory to be read out of bounds. | ||
|---|---|---|---|
| Submitted: | 8 Jan 11:54 | Modified: | 10 Jan 8:47 |
| Reporter: | fan devin | Email Updates: | |
| Status: | Can't repeat | Impact on me: | |
| Category: | MySQL Server: DDL | Severity: | S2 (Serious) |
| Version: | 8.0 | OS: | Any |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | Online DDL | ||
[8 Jan 16:46]
fan devin
There are some errors in the previous comments. If you want to repeat the problem, you should add ut_a (mrec < mrec_end) after mrec += extra_size at storage/innobase/row/row0log.cc：2467. Otherwise, it is difficult to cause a segfault, because although the read address is out of bounds, it may be valid allocated elsewhere.
[9 Jan 16:17]
MySQL Verification Team
Hi Mr. devin, Thank you for your bug report. However, we can not repeat it. Simply, innodb_sort_buffer_size is a read-only variable. You should send us a mysql-test/ script or stand-alone test case that we can reproduce. Hence , your test case is not complete. Can't repeat.
[10 Jan 8:38]
fan devin
I rewrote a test case that can be reproduced. use: mtr innodb-table-online-delete-release --valgrind --record
Attachment: test_case.zip (application/zip, text), 1.02 KiB.
[10 Jan 8:41]
fan devin
I used the above script to reproduce the problem in 8.0.33. core stack: ==31530== Thread 40 connection: ==31530== Invalid read of size 1 ==31530== at 0x26CA300: rec_init_offsets_comp_ordinary(unsigned char const*, bool, dict_index_t const*, unsigned long*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x25FC777: row_log_table_apply_op(que_thr_t*, unsigned long, unsigned long, ddl::Dup*, dberr_t*, mem_block_info_t*, mem_block_info_t*, unsigned char const*, unsigned char const*, unsigned long*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x25FFEAB: row_log_table_apply_ops(que_thr_t*, ddl::Dup*, Alter_stage*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x26022A7: row_log_table_apply(que_thr_t*, dict_table_t*, TABLE*, Alter_stage*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x24B5179: bool ha_innobase::inplace_alter_table_impl<dd::Table>(TABLE*, Alter_inplace_info*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x24B53EE: ha_innobase::inplace_alter_table(TABLE*, Alter_inplace_info*, dd::Table const*, dd::Table*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xF51546: mysql_inplace_alter_table(THD*, dd::Schema const&, dd::Schema const&, dd::Table const*, dd::Table*, Table_ref*, TABLE*, TABLE*, Alter_inplace_info*, enum_alter_inplace_result, Alter_table_ctx*, std::set<std::string, std::less<std::string>, Stateless_allocator<std::string, histograms::Histogram_psi_key_alloc, My_free_functor> >&, FOREIGN_KEY*, unsigned int, Foreign_key_parents_invalidator*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xF66B7D: mysql_alter_table(THD*, char const*, char const*, HA_CREATE_INFO*, Table_ref*, Alter_info*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x13B6B97: Sql_cmd_alter_table::execute(THD*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xEA3FE1: mysql_execute_command(THD*, bool) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xEA8C83: dispatch_sql_command(THD*, Parser_state*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xEA9F4C: dispatch_command(THD*, COM_DATA const*, enum_server_command) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xEAC17E: do_command(THD*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x1017B77: handle_connection (in /usr/local/mysql/bin/mysqld) ==31530== by 0x2BF2A64: pfs_spawn_thread (in /usr/local/mysql/bin/mysqld) ==31530== by 0x5960EA4: start_thread (in /usr/lib64/libpthread-2.17.so) ==31530== Address 0x4d2f002 is not stack'd, malloc'd or (recently) free'd ==31530== 2025-01-10T08:20:54Z UTC - mysqld got signal 11 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. ... Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (14ad4a70): alter table my_table engine=innodb Connection ID (thread ID): 9 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file ==31530== ==31530== Process terminating with default action of signal 11 (SIGSEGV): dumping core ==31530== at 0x5965AA1: pthread_kill (in /usr/lib64/libpthread-2.17.so) ==31530== by 0x102777C: handle_fatal_signal (in /usr/local/mysql/bin/mysqld) ==31530== by 0x596862F: ??? (in /usr/lib64/libpthread-2.17.so) ==31530== by 0x26CA2FF: rec_init_offsets_comp_ordinary(unsigned char const*, bool, dict_index_t const*, unsigned long*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x25FC777: row_log_table_apply_op(que_thr_t*, unsigned long, unsigned long, ddl::Dup*, dberr_t*, mem_block_info_t*, mem_block_info_t*, unsigned char const*, unsigned char const*, unsigned long*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x25FFEAB: row_log_table_apply_ops(que_thr_t*, ddl::Dup*, Alter_stage*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x26022A7: row_log_table_apply(que_thr_t*, dict_table_t*, TABLE*, Alter_stage*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x24B5179: bool ha_innobase::inplace_alter_table_impl<dd::Table>(TABLE*, Alter_inplace_info*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x24B53EE: ha_innobase::inplace_alter_table(TABLE*, Alter_inplace_info*, dd::Table const*, dd::Table*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xF51546: mysql_inplace_alter_table(THD*, dd::Schema const&, dd::Schema const&, dd::Table const*, dd::Table*, Table_ref*, TABLE*, TABLE*, Alter_inplace_info*, enum_alter_inplace_result, Alter_table_ctx*, std::set<std::string, std::less<std::string>, Stateless_allocator<std::string, histograms::Histogram_psi_key_alloc, My_free_functor> >&, FOREIGN_KEY*, unsigned int, Foreign_key_parents_invalidator*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xF66B7D: mysql_alter_table(THD*, char const*, char const*, HA_CREATE_INFO*, Table_ref*, Alter_info*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0x13B6B97: Sql_cmd_alter_table::execute(THD*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xEA3FE1: mysql_execute_command(THD*, bool) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xEA8C83: dispatch_sql_command(THD*, Parser_state*) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xEA9F4C: dispatch_command(THD*, COM_DATA const*, enum_server_command) (in /usr/local/mysql/bin/mysqld) ==31530== by 0xEAC17E: do_command(THD*) (in /usr/local/mysql/bin/mysqld)
[10 Jan 8:47]
fan devin
Note: you need to start mtr with --valgrind and innodb-table-online-delete-release-master.opt configurations, otherwise it will be very difficult to reproduce the problem.
Description: In the static const ddl::mrec_t *row_log_table_apply_op function, for the row_log of the ROW_T_INSERT and ROW_T_UPDATE types, the extra_info spans two blocks is avoided (returning nullptr), but for the row_log of the ROW_T_DELETE type, the pointer out-of-bounds problem is not judged, which may lead to Read out of bounds memory. // source code switch (*mrec++) { default: *error = DB_CORRUPTION; ut_d(ut_error); ut_o(return (nullptr)); case ROW_T_INSERT: extra_size = *mrec++; if (extra_size >= 0x80) { /* Read another byte of extra_size. */ extra_size = (extra_size & 0x7f) << 8; extra_size |= *mrec++; } mrec += extra_size; // insert op check here if (mrec > mrec_end) { return (nullptr); } ... ... ... case ROW_T_DELETE: /* 1 (extra_size) + at least 1 (payload) */ if (mrec + 2 >= mrec_end) { return (nullptr); } extra_size = *mrec++; ut_ad(mrec < mrec_end); /* We assume extra_size < 0x100 for the PRIMARY KEY prefix. For fixed-length PRIMARY key columns, it is 0. */ mrec += extra_size; //delete op not check mrec out of bound // will use illegal address in rec_deserialize_init_offsets rec_offs_set_n_fields(offsets, new_index->n_uniq + 2); rec_deserialize_init_offsets(mrec, new_index, offsets); next_mrec = mrec + rec_offs_data_size(offsets); if (log->table->n_v_cols) { if (next_mrec + 2 > mrec_end) { return (nullptr); } next_mrec += mach_read_from_2(next_mrec); } How to repeat: mtr case: start --mysqld=--innodb-sort-buffer-size=65536 --source include/have_debug_sync.inc use test; CREATE TABLE my_table ( ID int NOT NULL AUTO_INCREMENT, NAME varchar(200) NOT NULL, TIME timestamp DEFAULT CURRENT_TIMESTAMP, NAME2 varchar(64) NOT NULL, NAME3 varchar(64) NOT NULL, value int NOT NULL, PRIMARY KEY (ID, NAME, TIME, NAME2, NAME3) ); DELIMITER $$; CREATE PROCEDURE InsertData() BEGIN DECLARE i INT DEFAULT 0; WHILE i < 10 DO INSERT INTO my_table (NAME, TIME, NAME2, NAME3, value) SELECT CONCAT('Name_', FLOOR(RAND() * 1000)) AS NAME, NOW() AS TIME, CONCAT('Name2_', FLOOR(RAND() * 100)) AS NAME2, CONCAT('Name3_', FLOOR(RAND() * 100)) AS NAME3, FLOOR(RAND() * 100) AS value FROM information_schema.COLUMNS LIMIT 3000; SET i = i + 1; END WHILE; END$$ DELIMITER ;$$ call InsertData; connect (con1,localhost,root,,); connection con1; use test; SET DEBUG_SYNC = 'row_log_table_apply1_before signal rebuild wait_for delete'; --send alter table my_table engine=innodb connection default; delete from my_table; SET DEBUG_SYNC= 'now wait_for rebuild'; SET DEBUG_SYNC= 'now signal delete'; connection con1; --reap connection default; drop table my_table; drop procedure InsertData; disconnect con1; =================== It does not cause a crash every time, because although an out-of-bounds read occurs, the memory read may have been allocated. Even if the wrong value is read, it wiil check out-of-memory in subsequent processing(if (next_mrec > mrec_end) return nullptr). It is best to add ut_a(mrec < mrec_end) after extra_size = *mrec++; in the code, or use valgrind, the exception will reappear quickly. // source code extra_size = *mrec++; ut_ad(mrec < mrec_end); mrec += extra_size; rec_offs_set_n_fields(offsets, new_index->n_uniq + 2); rec_deserialize_init_offsets(mrec, new_index, offsets); next_mrec = mrec + rec_offs_data_size(offsets); if (log->table->n_v_cols) { if (next_mrec + 2 > mrec_end) { return (nullptr); } next_mrec += mach_read_from_2(next_mrec); } // Even if the wrong value is read, after adding it, nullptr will be returned. if (next_mrec > mrec_end) { return (nullptr); } Suggested fix: add mrec out of bounds check in row_log_table_apply_op case ROW_T_DELETE: /* 1 (extra_size) + at least 1 (payload) */ if (mrec + 2 >= mrec_end) { return (nullptr); } extra_size = *mrec++; ut_ad(mrec < mrec_end); /* We assume extra_size < 0x100 for the PRIMARY KEY prefix. For fixed-length PRIMARY key columns, it is 0. */ mrec += extra_size; + if (mrec > mrec_end) { + return (nullptr); + } rec_offs_set_n_fields(offsets, new_index->n_uniq + 2); /* info bits is not recorded, see row_log_table_delete for more information. In fact, the new index has no instantly modified column, so in this function, the last param of rec_deserialize_init_offsets can be true or false(both are ok). */ rec_deserialize_init_offsets(mrec, new_index, offsets, true); next_mrec = mrec + rec_offs_data_size(offsets); if (log->table->n_v_cols) { if (next_mrec + 2 > mrec_end) { return (nullptr); }