Bug #96472 Memory leak after 'innodb.alter_crash'
Submitted: 8 Aug 2019 13:05 Modified: 8 Aug 2019 16:24
Reporter: Yura Sorokin (OCA) Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S7 (Test Cases)
Version:5.7.27 OS:Any
Assigned to: CPU Architecture:Any

[8 Aug 2019 13:05] Yura Sorokin
Description:
The following Address Sanitizer log is generated after running 'innodb.alter_crash' MTR test case

Indirect leak of 2208 byte(s) in 1 object(s) allocated from:
    #0 0x7f26f4429f00 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xedf00)
    #1 0x558b89812179 in ut_allocator<unsigned char>::allocate(unsigned long, unsigned char const*, char const*, bool, bool) /home/yura/addon/local/percona-server/storage/innobase/include/ut0new.h:359
    #2 0x558b89812179 in mem_heap_create_block_func(mem_block_info_t*, unsigned long, char const*, unsigned long, unsigned long) /home/yura/addon/local/percona-server/storage/innobase/mem/mem0mem.cc:302
    #3 0x558b89813519 in mem_heap_add_block(mem_block_info_t*, unsigned long) /home/yura/addon/local/percona-server/storage/innobase/mem/mem0mem.cc:407
    #4 0x558b89813a18 in mem_heap_alloc /home/yura/addon/local/percona-server/storage/innobase/include/mem0mem.ic:204
    #5 0x558b89813a18 in mem_heap_dup(mem_block_info_t*, void const*, unsigned long) /home/yura/addon/local/percona-server/storage/innobase/mem/mem0mem.cc:59
    #6 0x558b89813bb0 in mem_heap_strdup(mem_block_info_t*, char const*) /home/yura/addon/local/percona-server/storage/innobase/mem/mem0mem.cc:46
    #7 0x558b89e12640 in dict_mem_fill_index_struct /home/yura/addon/local/percona-server/storage/innobase/include/dict0mem.ic:51
    #8 0x558b89e12640 in dict_mem_index_create(char const*, char const*, unsigned long, unsigned long, unsigned long) /home/yura/addon/local/percona-server/storage/innobase/dict/dict0mem.cc:692
    #9 0x558b8996439f in row_merge_create_index(trx_t*, dict_table_t*, index_def_t const*, dict_add_v_col_t const*) /home/yura/addon/local/percona-server/storage/innobase/row/row0merge.cc:4282
    #10 0x558b896ea4cc in prepare_inplace_alter_table_dict /home/yura/addon/local/percona-server/storage/innobase/handler/handler0alter.cc:4963
    #11 0x558b896f91ec in ha_innobase::prepare_inplace_alter_table(TABLE*, Alter_inplace_info*) /home/yura/addon/local/percona-server/storage/innobase/handler/handler0alter.cc:6382
    #12 0x558b876b81dd in handler::ha_prepare_inplace_alter_table(TABLE*, Alter_inplace_info*) /home/yura/addon/local/percona-server/sql/handler.cc:5140
    #13 0x558b88b5c58c in mysql_inplace_alter_table /home/yura/addon/local/percona-server/sql/sql_table.cc:7838
    #14 0x558b88b75d0b in mysql_alter_table(THD*, char const*, char const*, st_ha_create_information*, TABLE_LIST*, Alter_info*) /home/yura/addon/local/percona-server/sql/sql_table.cc:10317
    #15 0x558b890b8809 in Sql_cmd_alter_table::execute(THD*) /home/yura/addon/local/percona-server/sql/sql_alter.cc:348
    #16 0x558b889ca5d3 in mysql_execute_command(THD*, bool) /home/yura/addon/local/percona-server/sql/sql_parse.cc:5136
    #17 0x558b889d2535 in mysql_parse(THD*, Parser_state*, bool) /home/yura/addon/local/percona-server/sql/sql_parse.cc:5905
    #18 0x558b889d77d2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/addon/local/percona-server/sql/sql_parse.cc:1532
    #19 0x558b889ddbd0 in do_command(THD*) /home/yura/addon/local/percona-server/sql/sql_parse.cc:1053
    #20 0x558b88cdda44 in handle_connection /home/yura/addon/local/percona-server/sql/conn_handler/connection_handler_per_thread.cc:318
    #21 0x558b89471607 in pfs_spawn_thread /home/yura/addon/local/percona-server/storage/perfschema/pfs.cc:2190
    #22 0x7f26f41246da in start_thread /build/glibc-OTsEL5/glibc-2.27/nptl/pthread_create.c:463

How to repeat:
build MySQL Server with gcc-8 and with Address Sanitizer enabled (-DWITH_ASAN=ON)
CC=gcc-8 CXX=g++-8 cmake -DWITH_ASAN=ON ...

run
./mysql-test/mtr --big-test --debug-server --sanitize innodb.alter_crash
[8 Aug 2019 16:24] MySQL Verification Team
Thank you for the bug report.
[12 Aug 2019 13:19] Erlend Dahl
Posted by developer:
 
Repeatable with clang 7 on recent 5.7. Not repeatable on 8.0.
[14 Jan 13:16] Marcelo Altmann
If online alter fails to allocate undo log, heap memory allocated to the index object won't be freed. This can be simulated by using debug point 'ib_create_table_fail_too_many_trx' prior to an alter table.

To fix it, prior to set index pointer to null at row_merge_create_index, we should call dict_mem_index_free to correctly free memory objects allocated by the index.
[14 Jan 13:17] Marcelo Altmann
Suggested fix:

diff --git a/storage/innobase/row/row0merge.cc b/storage/innobase/row/row0merge.cc
index 5d0919a..c626c50 100644
--- a/storage/innobase/row/row0merge.cc
+++ b/storage/innobase/row/row0merge.cc
@@ -4209,6 +4209,7 @@ row_merge_create_index(
                this index, to ensure read consistency. */
                ut_ad(index->trx_id == trx->id);
        } else {
+               dict_mem_index_free(index);
                index = NULL;
        }
[17 Jan 11:40] Marcelo Altmann
(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.
[8 Apr 13:49] Marcelo Altmann
an enhanced version of the fix to avoid double free:

diff --git a/storage/innobase/row/row0merge.cc b/storage/innobase/row/row0merge.cc
index 5d0919a322b..29f2ea993f1 100644
--- a/storage/innobase/row/row0merge.cc
+++ b/storage/innobase/row/row0merge.cc
@@ -4209,6 +4209,11 @@ row_merge_create_index(
                this index, to ensure read consistency. */
                ut_ad(index->trx_id == trx->id);
        } else {
+               /* In case we were unable to assign an undo record for this index
+               we won't free index memory object */
+               if (err == DB_TOO_MANY_CONCURRENT_TRXS) {
+                       dict_mem_index_free(index);
+               }
                index = NULL;
        }

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.