Description:
A crash occurs when mysqld hosting a large number of tables is stopped and started multiple times in a loop. The crash occurs during mysqld startup, the immediate cause is that mysqld receives signal 6. This was observed on mysql-trunk 0edbcfd6, but could have been introduced no more than a week earlier (the previous weekly test did not trigger the problem).
The following stack trace is printed to the console from which mysqld was started:
*** Error in `/.../mysqld': double free or corruption (fasttop): 0x00007f5f5c44ecc0 ***
*** Error in `/.../mysqld': double free or corruption (fasttop): 0x00007f63ca2d7970 ***
======= Backtrace: =========
======= Backtrace: =========
/lib64/libc.so.6(+0x7c619)[0x7f63e0a2d619]
/lib64/libc.so.6(+0x7c619)[0x7f63e0a2d619]
mysqld(_Z21log_sink_buffer_flush26log_sink_buffer_flush_mode+0x227)[0x2212ab7]
mysqld(_Z21log_sink_buffer_flush26log_sink_buffer_flush_mode+0x227)[0x2212ab7]
mysqld[0x2212ce5]
mysqld[0x2212ce5]
mysqld(_Z15log_line_submitP9_log_line+0x3c0)[0x2212670]
mysqld(_Z15log_line_submitP9_log_line+0x3c0)[0x2212670]
mysqld(_ZN2ib6loggerD1Ev+0x149)[0x1e777e9]
mysqld(_ZN2ib6loggerD1Ev+0x149)[0x1e777e9]
mysqld(_ZN14Validate_files5checkERKN9__gnu_cxx17__normal_iteratorIPKPKN2dd10TablespaceESt6vectorIS5_SaIS5_EEEESD_mPm+0x485)[0x1c9d915]
mysqld(_ZN14Validate_files5checkERKN9__gnu_cxx17__normal_iteratorIPKPKN2dd10TablespaceESt6vectorIS5_SaIS5_EEEESD_mPm+0x485)[0x1c9d915]
mysqld(_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJ8RunnableSt8functionIFvRKN9__gnu_cxx17__normal_iteratorIPKPKN2dd10TablespaceESt6vectorISA_SaISA_EEEESI_mPmEESG_SG_mSJ_EEEEE6_M_runEv+0xe0)[0x1c8da70]
mysqld(_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJ8RunnableSt8functionIFvRKN9__gnu_cxx17__normal_iteratorIPKPKN2dd10TablespaceESt6vectorISA_SaISA_EEEESI_mPmEESG_SG_mSJ_EEEEE6_M_runEv+0xe0)[0x1c8da70]
Attached to the report are:
- reproduction scripts
- mysqld core dump
- mysqld start up log (with stack trace)
How to repeat:
1. Initialize mysqld and create 500k tables using one of the attached scripts (this requires ca. 36 GB of free space; I wasn't able to repro with a smaller number of tables):
test_table_creation_throughput.sh -b <path to mysql basedir> -d <path to datadir> -t 10 -i 100 -j 500 -m mix
2. Run a loop of start/stop mysqld using one of the attached scripts:
measure_time_to_start_server.sh -b <path to mysql basedir> -d <path to datadir> -l 10
In my testing the crash occurs around the third start.
