Bug #75643 mysql crashed on dict_stats_update
Submitted: 27 Jan 2015 8:32 Modified: 13 Apr 0:08
Reporter: zhai weixiang (OCA) Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S3 (Non-critical)
Version:5.6.16 OS:Any
Assigned to: CPU Architecture:Any

[27 Jan 2015 8:32] zhai weixiang
Description:
the crash described bellow is not repeatable and only appeared on production environment only once. It seems nobody has reported a similar one, I decided to file a bug here. 

Currently we are using 5.6.16 on production. It's heavily patched. but the code related to crash is not changed.

quoted from error log.

2014-10-21 10:46:10 2b35efd8f700  InnoDB: Assertion failure in thread 47510657234688 in file dict0stats.cc line 2902
InnoDB: Failing assertion: strchr(table->name, '/') != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
02:46:10 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=1048576
max_used_connections=3
max_threads=1550
thread_count=3
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4799332 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/u01/mysql_20140715/bin/mysqld(my_print_stacktrace+0x35)[0x8de285]
/u01/mysql_20140715/bin/mysqld(handle_fatal_signal+0x42b)[0x65129b]
/lib64/libpthread.so.0[0x313a60f500]
/lib64/libc.so.6(gsignal+0x35)[0x313a2328a5]
/lib64/libc.so.6(abort+0x175)[0x313a234085]
/u01/mysql_20140715/bin/mysqld[0xa6fb3f]
/u01/mysql_20140715/bin/mysqld[0xa7050a]
/lib64/libpthread.so.0[0x313a607851]
/lib64/libc.so.6(clone+0x6d)[0x313a2e767d]

How to repeat:
I don't know. The crash happened several months ago. We just found it recently from error log.

Suggested fix:
I don't know.
[27 Jan 2015 8:34] zhai weixiang
backtrace

Attachment: backtrace.txt (text/plain), 15.80 KiB.

[28 Jan 2015 8:30] Shane Bester
It looks like SQL thread (thread 7) was running OPTIMIZE TABLE, maybe that is a clue to get a repeatable test?  Any idea which table it was?
[28 Jan 2015 13:17] zhai weixiang
May be this one: ibayapp/#sql2-184ae-1a. it's ibd file is left in the data dir while crash happened.
[18 Sep 2015 17:53] Justin Tolmer
Possibly related to this bug, our test infrastructure hit a valgrind failure on that line of code in dict_stats_update running against a 5.6.26-based source tree. The exact source code used to build and run the tests is from https://github.com/webscalesql/webscalesql-5.6/tree/30d93217019b0c595b6016d173cd1612edce8d....

parts.partition_mgm_lc1_innodb           w14 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2015-09-18 02:15:53
line
==1292686== Thread 17:
==1292686== Invalid read of size 1
==1292686==    at 0x4C2E2B4: index (in /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==1292686==    by 0xD1A6B9: dict_stats_update(dict_table_t*, dict_stats_upd_option_t) (dict0stats.cc:3117)
==1292686==    by 0xD1E830: dict_stats_process_entry_from_recalc_pool() (dict0stats_bg.cc:313)
==1292686==    by 0xD1E90F: dict_stats_thread (dict0stats_bg.cc:355)
==1292686==    by 0x5291FA7: start_thread (in /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/libpthread-2.17.so)
==1292686==    by 0x6F4A5BC: clone (in /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/libc-2.17.so)
==1292686==  Address 0xdf17358 is 8 bytes inside a block of size 34 free'd
==1292686==    at 0x4C2D66E: realloc (in /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==1292686==    by 0xC68A78: ut_realloc(void*, unsigned long) (ut0mem.cc:284)
==1292686==    by 0xCF2F87: dict_table_rename_in_cache(dict_table_t*, char const*, unsigned long) (dict0dict.cc:1701)
==1292686==    by 0xBD4FB5: row_rename_table_for_mysql(char const*, char const*, trx_t*, bool) (row0mysql.cc:5091)
==1292686==    by 0xAD06DD: innobase_rename_table(trx_t*, char const*, char const*) (ha_innodb.cc:10345)
==1292686==    by 0xAD0992: ha_innobase::rename_table(char const*, char const*) (ha_innodb.cc:10448)
==1292686==    by 0x62D2B6: handler::ha_rename_table(char const*, char const*) (handler.cc:4530)
==1292686==    by 0xE1873C: ha_partition::del_ren_table(char const*, char const*) (ha_partition.cc:2347)
==1292686==    by 0xE1400A: ha_partition::rename_table(char const*, char const*) (ha_partition.cc:615)
==1292686==    by 0x62D2B6: handler::ha_rename_table(char const*, char const*) (handler.cc:4530)
==1292686==    by 0x83A58F: mysql_rename_table(handlerton*, char const*, char const*, char const*, char const*, unsigned int) (sql_table.cc:5249)
==1292686==    by 0x842707: mysql_alter_table(THD*, char*, char*, st_ha_create_information*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool) (sql_table.cc:8665)
==1292686==    by 0x98CC59: Sql_cmd_alter_table::execute(THD*) (sql_alter.cc:313)
==1292686==    by 0x7CF668: mysql_execute_command(THD*) (sql_parse.cc:5093)
==1292686==    by 0x7D2FA8: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6619)
==1292686==    by 0x7C5B28: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1440)
[28 Jun 2017 5:52] Daniël van Eeden
This happened to me with 5.7.17
Stacktrace:
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf42a95]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7cd8b4]
/lib64/libpthread.so.0(+0xf7e0)[0x7f80d77587e0]
/lib64/libc.so.6(gsignal+0x35)[0x7f80d61f85e5]
/lib64/libc.so.6(abort+0x175)[0x7f80d61f9dc5]
/usr/lib64/libstdc++.so.6(__gnu_cxx::__verbose_terminate_handler()+0x12d)[0x7f80d6ab2a7d]
/usr/lib64/libstdc++.so.6(+0xbcbd6)[0x7f80d6ab0bd6]
/usr/lib64/libstdc++.so.6(+0xbcc03)[0x7f80d6ab0c03]
/usr/lib64/libstdc++.so.6(+0xbcd22)[0x7f80d6ab0d22]
/usr/lib64/libstdc++.so.6(std::__throw_out_of_range(char const*)+0x67)[0x7f80d6a55db7]
/usr/sbin/mysqld[0x121fbd8]
/usr/sbin/mysqld[0x122070f]
/usr/sbin/mysqld(dict_stats_update(dict_table_t*, dict_stats_upd_option_t)+0x923)[0x12252f3]
/usr/sbin/mysqld(dict_stats_thread+0x53a)[0x1227a9a]
/lib64/libpthread.so.0(+0x7aa1)[0x7f80d7750aa1]
/lib64/libc.so.6(clone+0x6d)[0x7f80d62aeaad]
[28 Jun 2017 5:59] Daniël van Eeden
Looks like this is fixed in 5.7.18
[28 Jun 2017 6:11] Shane Bester
Daniel,  your crash was fixed in 5.7.18, see

https://bugs.mysql.com/84940
Bug 24585978 - INNODB: ASSERTION TOTAL_RECS > 0 FAILURE IN FILE DICT0STATS.CC

Zhai's crash is slightly different and reported in early 2015 before 84940 existed..
[13 Mar 0:08] Miguel Solorzano
Since the crash was observed once and doesn't has a repeatable test case, question: it was again noticed with most recent releases?. Thanks in advance.
[13 Apr 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".