Bug #117135 When executing online ddl rebuilt table, concurrent delete operations may cause row_log memory to be read out of bounds.
Submitted: 8 Jan 11:54 Modified: 10 Jan 12:03
Reporter: fan devin Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: DDL Severity:S2 (Serious)
Version:8.0 OS:Any
Assigned to: CPU Architecture:Any
Tags: Online DDL

[8 Jan 11:54] fan devin
Description:
In the static const ddl::mrec_t *row_log_table_apply_op function, for the row_log of the ROW_T_INSERT and ROW_T_UPDATE types, the extra_info spans two blocks is avoided (returning nullptr), but for the row_log of the ROW_T_DELETE type, the pointer out-of-bounds problem is not judged, which may lead to Read out of bounds memory.

// source code 

  switch (*mrec++) {
    default:
      *error = DB_CORRUPTION;
      ut_d(ut_error);
      ut_o(return (nullptr));
    case ROW_T_INSERT:
      extra_size = *mrec++;

      if (extra_size >= 0x80) {
        /* Read another byte of extra_size. */

        extra_size = (extra_size & 0x7f) << 8;
        extra_size |= *mrec++;
      }

      mrec += extra_size;

      // insert op check here

      if (mrec > mrec_end) {
        return (nullptr);
      }
...
...
...

    case ROW_T_DELETE:
      /* 1 (extra_size) + at least 1 (payload) */
      if (mrec + 2 >= mrec_end) {
        return (nullptr);
      }

      extra_size = *mrec++;
      ut_ad(mrec < mrec_end);

      /* We assume extra_size < 0x100 for the PRIMARY KEY prefix.
      For fixed-length PRIMARY key columns, it is 0. */
      mrec += extra_size;
      
      //delete op not check mrec out of bound
      // will use illegal address in rec_deserialize_init_offsets

      rec_offs_set_n_fields(offsets, new_index->n_uniq + 2);
      rec_deserialize_init_offsets(mrec, new_index, offsets);
      next_mrec = mrec + rec_offs_data_size(offsets);
      if (log->table->n_v_cols) {
        if (next_mrec + 2 > mrec_end) {
          return (nullptr);
        }

        next_mrec += mach_read_from_2(next_mrec);
      }

How to repeat:
mtr case: start  --mysqld=--innodb-sort-buffer-size=65536

--source include/have_debug_sync.inc

use test;
CREATE TABLE my_table (
    ID int NOT NULL AUTO_INCREMENT,
    NAME varchar(200) NOT NULL,
    TIME timestamp DEFAULT CURRENT_TIMESTAMP,
    NAME2 varchar(64) NOT NULL,
    NAME3 varchar(64) NOT NULL,
    value int NOT NULL,
    PRIMARY KEY (ID, NAME, TIME, NAME2, NAME3)
);

DELIMITER $$;

CREATE PROCEDURE InsertData()
BEGIN
    DECLARE i INT DEFAULT 0;

    WHILE i < 10 DO
        INSERT INTO my_table (NAME, TIME, NAME2, NAME3, value)
        SELECT 
            CONCAT('Name_', FLOOR(RAND() * 1000)) AS NAME,
            NOW() AS TIME,
            CONCAT('Name2_', FLOOR(RAND() * 100)) AS NAME2, 
            CONCAT('Name3_', FLOOR(RAND() * 100)) AS NAME3,
            FLOOR(RAND() * 100) AS value 
        FROM 
            information_schema.COLUMNS 
        LIMIT 3000;

        SET i = i + 1;
    END WHILE;
END$$

DELIMITER ;$$
call InsertData;

connect (con1,localhost,root,,);
connection con1;
use test;
SET DEBUG_SYNC = 'row_log_table_apply1_before signal rebuild wait_for delete';
--send alter table my_table engine=innodb

connection default;
delete from my_table;
SET DEBUG_SYNC= 'now wait_for rebuild';
SET DEBUG_SYNC= 'now signal delete';

connection con1;
--reap

connection default;
drop table my_table;
drop procedure InsertData;
disconnect con1;

===================
It does not cause a crash every time, because although an out-of-bounds read occurs, the memory read may have been allocated. Even if the wrong value is read, it wiil check out-of-memory in subsequent processing(if (next_mrec > mrec_end) return nullptr).

It is best to add ut_a(mrec < mrec_end) after extra_size = *mrec++; in the code, or use valgrind, the exception will reappear quickly.

// source code

      extra_size = *mrec++;
      ut_ad(mrec < mrec_end);
      mrec += extra_size;

      rec_offs_set_n_fields(offsets, new_index->n_uniq + 2);
      rec_deserialize_init_offsets(mrec, new_index, offsets);
      next_mrec = mrec + rec_offs_data_size(offsets);
      if (log->table->n_v_cols) {
        if (next_mrec + 2 > mrec_end) {
          return (nullptr);
        }

        next_mrec += mach_read_from_2(next_mrec);
      }

      // Even if the wrong value is read, after adding it, nullptr will be returned.

      if (next_mrec > mrec_end) {
        return (nullptr);
      }

Suggested fix:
add mrec out of bounds check in row_log_table_apply_op

    case ROW_T_DELETE:
      /* 1 (extra_size) + at least 1 (payload) */
      if (mrec + 2 >= mrec_end) {
        return (nullptr);
      }

      extra_size = *mrec++;
      ut_ad(mrec < mrec_end);

      /* We assume extra_size < 0x100 for the PRIMARY KEY prefix.
      For fixed-length PRIMARY key columns, it is 0. */
      mrec += extra_size;

+     if (mrec > mrec_end) {
+       return (nullptr);
+     }

      rec_offs_set_n_fields(offsets, new_index->n_uniq + 2);
      /* info bits is not recorded, see row_log_table_delete for more
      information. In fact, the new index has no instantly modified column,
      so in this function, the last param of rec_deserialize_init_offsets
      can be true or false(both are ok). */
      rec_deserialize_init_offsets(mrec, new_index, offsets, true);
      next_mrec = mrec + rec_offs_data_size(offsets);
      if (log->table->n_v_cols) {
        if (next_mrec + 2 > mrec_end) {
          return (nullptr);
        }
[8 Jan 16:46] fan devin
There are some errors in the previous comments. If you want to repeat the problem, you should add ut_a (mrec < mrec_end) after mrec += extra_size at storage/innobase/row/row0log.cc:2467. Otherwise, it is difficult to cause a segfault, because although the read address is out of bounds, it may be valid allocated elsewhere.
[9 Jan 16:17] MySQL Verification Team
Hi Mr. devin,

Thank you for your bug report.

However, we can not repeat it.

Simply, innodb_sort_buffer_size is a read-only variable. 

You should send us a mysql-test/ script or stand-alone test case that we can reproduce.

Hence , your test case is not complete.

Can't repeat.
[10 Jan 8:38] fan devin
I rewrote a test case that can be reproduced. use: mtr innodb-table-online-delete-release --valgrind --record

Attachment: test_case.zip (application/zip, text), 1.02 KiB.

[10 Jan 8:41] fan devin
I used the above script to reproduce the problem in 8.0.33.
core stack:
==31530== Thread 40 connection:
==31530== Invalid read of size 1
==31530==    at 0x26CA300: rec_init_offsets_comp_ordinary(unsigned char const*, bool, dict_index_t const*, unsigned long*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x25FC777: row_log_table_apply_op(que_thr_t*, unsigned long, unsigned long, ddl::Dup*, dberr_t*, mem_block_info_t*, mem_block_info_t*, unsigned char const*, unsigned char const*, unsigned long*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x25FFEAB: row_log_table_apply_ops(que_thr_t*, ddl::Dup*, Alter_stage*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x26022A7: row_log_table_apply(que_thr_t*, dict_table_t*, TABLE*, Alter_stage*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x24B5179: bool ha_innobase::inplace_alter_table_impl<dd::Table>(TABLE*, Alter_inplace_info*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x24B53EE: ha_innobase::inplace_alter_table(TABLE*, Alter_inplace_info*, dd::Table const*, dd::Table*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xF51546: mysql_inplace_alter_table(THD*, dd::Schema const&, dd::Schema const&, dd::Table const*, dd::Table*, Table_ref*, TABLE*, TABLE*, Alter_inplace_info*, enum_alter_inplace_result, Alter_table_ctx*, std::set<std::string, std::less<std::string>, Stateless_allocator<std::string, histograms::Histogram_psi_key_alloc, My_free_functor> >&, FOREIGN_KEY*, unsigned int, Foreign_key_parents_invalidator*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xF66B7D: mysql_alter_table(THD*, char const*, char const*, HA_CREATE_INFO*, Table_ref*, Alter_info*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x13B6B97: Sql_cmd_alter_table::execute(THD*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xEA3FE1: mysql_execute_command(THD*, bool) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xEA8C83: dispatch_sql_command(THD*, Parser_state*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xEA9F4C: dispatch_command(THD*, COM_DATA const*, enum_server_command) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xEAC17E: do_command(THD*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x1017B77: handle_connection (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x2BF2A64: pfs_spawn_thread (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x5960EA4: start_thread (in /usr/lib64/libpthread-2.17.so)
==31530==  Address 0x4d2f002 is not stack'd, malloc'd or (recently) free'd
==31530==
2025-01-10T08:20:54Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
...
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (14ad4a70): alter table my_table engine=innodb
Connection ID (thread ID): 9
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
==31530==
==31530== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==31530==    at 0x5965AA1: pthread_kill (in /usr/lib64/libpthread-2.17.so)
==31530==    by 0x102777C: handle_fatal_signal (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x596862F: ??? (in /usr/lib64/libpthread-2.17.so)
==31530==    by 0x26CA2FF: rec_init_offsets_comp_ordinary(unsigned char const*, bool, dict_index_t const*, unsigned long*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x25FC777: row_log_table_apply_op(que_thr_t*, unsigned long, unsigned long, ddl::Dup*, dberr_t*, mem_block_info_t*, mem_block_info_t*, unsigned char const*, unsigned char const*, unsigned long*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x25FFEAB: row_log_table_apply_ops(que_thr_t*, ddl::Dup*, Alter_stage*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x26022A7: row_log_table_apply(que_thr_t*, dict_table_t*, TABLE*, Alter_stage*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x24B5179: bool ha_innobase::inplace_alter_table_impl<dd::Table>(TABLE*, Alter_inplace_info*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x24B53EE: ha_innobase::inplace_alter_table(TABLE*, Alter_inplace_info*, dd::Table const*, dd::Table*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xF51546: mysql_inplace_alter_table(THD*, dd::Schema const&, dd::Schema const&, dd::Table const*, dd::Table*, Table_ref*, TABLE*, TABLE*, Alter_inplace_info*, enum_alter_inplace_result, Alter_table_ctx*, std::set<std::string, std::less<std::string>, Stateless_allocator<std::string, histograms::Histogram_psi_key_alloc, My_free_functor> >&, FOREIGN_KEY*, unsigned int, Foreign_key_parents_invalidator*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xF66B7D: mysql_alter_table(THD*, char const*, char const*, HA_CREATE_INFO*, Table_ref*, Alter_info*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0x13B6B97: Sql_cmd_alter_table::execute(THD*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xEA3FE1: mysql_execute_command(THD*, bool) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xEA8C83: dispatch_sql_command(THD*, Parser_state*) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xEA9F4C: dispatch_command(THD*, COM_DATA const*, enum_server_command) (in /usr/local/mysql/bin/mysqld)
==31530==    by 0xEAC17E: do_command(THD*) (in /usr/local/mysql/bin/mysqld)
[10 Jan 8:47] fan devin
Note: you need to start mtr with --valgrind and innodb-table-online-delete-release-master.opt configurations, otherwise it will be very difficult to reproduce the problem.
[10 Jan 12:03] MySQL Verification Team
Hi Mr. devin,

Thank you for your bug report.

After much effort, we have managed to repeat the bug.

Thank you very much.

Our stacktrace is very, very similar to yours.

This is now a verified bug report for version 8.0 and all higher versions.