MySQL Bugs: #75643: mysql crashed on dict_stats

Bug #75643	mysql crashed on dict_stats_update
Submitted:	27 Jan 2015 8:32	Modified:	13 Apr 0:08
Reporter:	zhai weixiang (OCA)	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S3 (Non-critical)
Version:	5.6.16	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
the crash described bellow is not repeatable and only appeared on production environment only once. It seems nobody has reported a similar one, I decided to file a bug here. 

Currently we are using 5.6.16 on production. It's heavily patched. but the code related to crash is not changed.

quoted from error log.

2014-10-21 10:46:10 2b35efd8f700  InnoDB: Assertion failure in thread 47510657234688 in file dict0stats.cc line 2902
InnoDB: Failing assertion: strchr(table->name, '/') != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
02:46:10 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=1048576
max_used_connections=3
max_threads=1550
thread_count=3
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4799332 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/u01/mysql_20140715/bin/mysqld(my_print_stacktrace+0x35)[0x8de285]
/u01/mysql_20140715/bin/mysqld(handle_fatal_signal+0x42b)[0x65129b]
/lib64/libpthread.so.0[0x313a60f500]
/lib64/libc.so.6(gsignal+0x35)[0x313a2328a5]
/lib64/libc.so.6(abort+0x175)[0x313a234085]
/u01/mysql_20140715/bin/mysqld[0xa6fb3f]
/u01/mysql_20140715/bin/mysqld[0xa7050a]
/lib64/libpthread.so.0[0x313a607851]
/lib64/libc.so.6(clone+0x6d)[0x313a2e767d]

How to repeat:
I don't know. The crash happened several months ago. We just found it recently from error log.

Suggested fix:
I don't know.

backtrace

Attachment: backtrace.txt (text/plain), 15.80 KiB.

It looks like SQL thread (thread 7) was running OPTIMIZE TABLE, maybe that is a clue to get a repeatable test?  Any idea which table it was?

May be this one: ibayapp/#sql2-184ae-1a. it's ibd file is left in the data dir while crash happened.

Possibly related to this bug, our test infrastructure hit a valgrind failure on that line of code in dict_stats_update running against a 5.6.26-based source tree. The exact source code used to build and run the tests is from https://github.com/webscalesql/webscalesql-5.6/tree/30d93217019b0c595b6016d173cd1612edce8d....

parts.partition_mgm_lc1_innodb           w14 [ fail ]  Found warnings/errors in server log file!
        Test ended at 2015-09-18 02:15:53
line
==1292686== Thread 17:
==1292686== Invalid read of size 1
==1292686==    at 0x4C2E2B4: index (in /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==1292686==    by 0xD1A6B9: dict_stats_update(dict_table_t*, dict_stats_upd_option_t) (dict0stats.cc:3117)
==1292686==    by 0xD1E830: dict_stats_process_entry_from_recalc_pool() (dict0stats_bg.cc:313)
==1292686==    by 0xD1E90F: dict_stats_thread (dict0stats_bg.cc:355)
==1292686==    by 0x5291FA7: start_thread (in /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/libpthread-2.17.so)
==1292686==    by 0x6F4A5BC: clone (in /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/libc-2.17.so)
==1292686==  Address 0xdf17358 is 8 bytes inside a block of size 34 free'd
==1292686==    at 0x4C2D66E: realloc (in /usr/local/fbcode/gcc-4.8.1-glibc-2.17/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==1292686==    by 0xC68A78: ut_realloc(void*, unsigned long) (ut0mem.cc:284)
==1292686==    by 0xCF2F87: dict_table_rename_in_cache(dict_table_t*, char const*, unsigned long) (dict0dict.cc:1701)
==1292686==    by 0xBD4FB5: row_rename_table_for_mysql(char const*, char const*, trx_t*, bool) (row0mysql.cc:5091)
==1292686==    by 0xAD06DD: innobase_rename_table(trx_t*, char const*, char const*) (ha_innodb.cc:10345)
==1292686==    by 0xAD0992: ha_innobase::rename_table(char const*, char const*) (ha_innodb.cc:10448)
==1292686==    by 0x62D2B6: handler::ha_rename_table(char const*, char const*) (handler.cc:4530)
==1292686==    by 0xE1873C: ha_partition::del_ren_table(char const*, char const*) (ha_partition.cc:2347)
==1292686==    by 0xE1400A: ha_partition::rename_table(char const*, char const*) (ha_partition.cc:615)
==1292686==    by 0x62D2B6: handler::ha_rename_table(char const*, char const*) (handler.cc:4530)
==1292686==    by 0x83A58F: mysql_rename_table(handlerton*, char const*, char const*, char const*, char const*, unsigned int) (sql_table.cc:5249)
==1292686==    by 0x842707: mysql_alter_table(THD*, char*, char*, st_ha_create_information*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool) (sql_table.cc:8665)
==1292686==    by 0x98CC59: Sql_cmd_alter_table::execute(THD*) (sql_alter.cc:313)
==1292686==    by 0x7CF668: mysql_execute_command(THD*) (sql_parse.cc:5093)
==1292686==    by 0x7D2FA8: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6619)
==1292686==    by 0x7C5B28: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1440)

This happened to me with 5.7.17
Stacktrace:
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf42a95]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7cd8b4]
/lib64/libpthread.so.0(+0xf7e0)[0x7f80d77587e0]
/lib64/libc.so.6(gsignal+0x35)[0x7f80d61f85e5]
/lib64/libc.so.6(abort+0x175)[0x7f80d61f9dc5]
/usr/lib64/libstdc++.so.6(__gnu_cxx::__verbose_terminate_handler()+0x12d)[0x7f80d6ab2a7d]
/usr/lib64/libstdc++.so.6(+0xbcbd6)[0x7f80d6ab0bd6]
/usr/lib64/libstdc++.so.6(+0xbcc03)[0x7f80d6ab0c03]
/usr/lib64/libstdc++.so.6(+0xbcd22)[0x7f80d6ab0d22]
/usr/lib64/libstdc++.so.6(std::__throw_out_of_range(char const*)+0x67)[0x7f80d6a55db7]
/usr/sbin/mysqld[0x121fbd8]
/usr/sbin/mysqld[0x122070f]
/usr/sbin/mysqld(dict_stats_update(dict_table_t*, dict_stats_upd_option_t)+0x923)[0x12252f3]
/usr/sbin/mysqld(dict_stats_thread+0x53a)[0x1227a9a]
/lib64/libpthread.so.0(+0x7aa1)[0x7f80d7750aa1]
/lib64/libc.so.6(clone+0x6d)[0x7f80d62aeaad]

Looks like this is fixed in 5.7.18

Daniel,  your crash was fixed in 5.7.18, see

https://bugs.mysql.com/84940
Bug 24585978 - INNODB: ASSERTION TOTAL_RECS > 0 FAILURE IN FILE DICT0STATS.CC

Zhai's crash is slightly different and reported in early 2015 before 84940 existed..

Since the crash was observed once and doesn't has a repeatable test case, question: it was again noticed with most recent releases?. Thanks in advance.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".