MySQL Bugs: #64913: crash after get_schema_tables_result when reading information

Bug #64913	crash after get_schema_tables_result when reading information_schema tables
Submitted:	9 Apr 2012 8:11	Modified:	3 Jul 2012 11:45
Reporter:	Simon Mudd (OCA)	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: Information schema	Severity:	S3 (Non-critical)
Version:	5.6.4-m7-log MySQL Community Server (GPL	OS:	Linux (centos-release-5-6.el5.centos.1)
Assigned to:		CPU Architecture:	Any
Tags:	windmill

Description:
Server crashed. Reporting in case this information is useful or has not been seen before.

The log file says:

120328 11:47:54 mysqld_safe mysqld from pid file /<path_to_datadir>/<servername>.pid ended
120328 11:53:56 mysqld_safe Starting mysqld_using_numactl daemon with databases from /<path_to_datadir>
120328 11:53:56 [Note] Plugin 'FEDERATED' is disabled.
120328 11:53:56 InnoDB: The InnoDB memory heap is disabled
120328 11:53:56 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120328 11:53:56 InnoDB: Compressed tables use zlib 1.2.3
120328 11:53:56 InnoDB: Using Linux native AIO
120328 11:53:56 InnoDB: CPU supports crc32 instructions
120328 11:53:56 InnoDB: Initializing buffer pool, size = 161.0G
120328 11:54:06 InnoDB: Completed initialization of buffer pool
120328 11:54:06 InnoDB: highest supported file format is Barracuda.
120328 11:54:11 InnoDB: 128 rollback segment(s) are active.
120328 11:54:11 InnoDB: Waiting for the background threads to start
120328 11:54:12 InnoDB: 1.2.4 started; log sequence number 176104831597
120328 11:54:12 [Note] Event Scheduler: Loaded 0 events
120328 11:54:12 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.4-m7-log'  socket: 'mysql.sock'  port: 3306  MySQL Community Server (GPL)
120328 11:54:21 [Note] Start binlog_dump to slave_server(187205015), pos(binlog.000064, 5236)
120328 11:54:21 [Note] Start binlog_dump to slave_server(187205014), pos(binlog.000064, 5236)
120403 10:34:00 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=805306368
read_buffer_size=4194304
max_used_connections=205
max_threads=3000
thread_count=20
connection_count=20
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 25398572 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x2ad5d9805d60
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4d3340a0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x8a4229]
/usr/sbin/mysqld(handle_segfault+0x3a1)[0x557a11]
/lib64/libpthread.so.0[0x31d340eb10]
/usr/sbin/mysqld[0x9c4063]
/usr/sbin/mysqld[0x93d25e]
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x26c)[0x70cc1c]
/usr/sbin/mysqld(_ZN4JOIN14prepare_resultEPP4ListI4ItemE+0x122)[0x6d57d2]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0xe1)[0x7007a1]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x2ad)[0x70970d]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x15e)[0x70988e]
/usr/sbin/mysqld[0x6b7a06]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4ba3)[0x6bcd83]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x294)[0x6be924]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x7e8)[0x6bf198]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xd7)[0x6c0617]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x10f)[0x69359f]
/usr/sbin/mysqld(handle_one_connection+0x45)[0x693685]
/usr/sbin/mysqld(pfs_spawn_thread+0x138)[0x9162f8]
/lib64/libpthread.so.0[0x31d340673d]
/lib64/libc.so.6(clone+0x6d)[0x31d2cd44bd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x2ad5d98eb480): is an invalid pointer
Connection ID (thread ID): 4
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120403 10:34:40 mysqld_safe Number of processes running now: 0

How to repeat:
N/A

Suggested fix:
Don't crash. More information will be provided directly to Oracle support.

mysqld_using_numactl is a shell wrapper to use numactl when starting mysqld. See: http://blog.wl0.org/2011/03/how-to-start-mysqld-using-numactl/

I got a repeatable crash when accessing information_schema tables in a join.
Without knowing the query involved in this crash, it's not possible to be 100% sure I got the same bug.

my 5.6.6 backtrace:
mysqld.exe!get_schema_tables_result()[sql_show.cc:7184]
mysqld.exe!JOIN::prepare_result()[sql_select.cc:795]
mysqld.exe!JOIN::exec()[sql_executor.cc:139]
mysqld.exe!mysql_select()[sql_select.cc:1069]
mysqld.exe!handle_select()[sql_select.cc:107]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:4929]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2488]
mysqld.exe!mysql_parse()[sql_parse.cc:6016]
mysqld.exe!dispatch_command()[sql_parse.cc:1260]
mysqld.exe!do_command()[sql_parse.cc:980]
mysqld.exe!do_handle_one_connection()[sql_connect.cc:942]
mysqld.exe!handle_one_connection()[sql_connect.cc:858]
mysqld.exe!pthread_start()[my_winthread.c:61]
mysqld.exe!_callthreadstartex()[threadex.c:314]
mysqld.exe!_threadstartex()[threadex.c:292]

Oracle bug number:  13966514

I had same crashes twice today, immediately after a deadlock detection.
My err file output is very similar, except it ends at this line:
stack_bottom = 0x7b8434eb8 thread_stack 0x40000
(and restart messages follow after)

Franjo, I've made you last comment private.  It has nothing to do with this crash
and didnt' want to it confusing the bug report.   Yours is InnoDB/Query Cache related, not information_schema related.

After the similar bug was fixed in 5.6.6:

Bug 13966514 - CRASH IN GET_SCHEMA_TABLES_RESULT WITH MIN/MAX, LEFT/RIGHT JOIN ON I_S TABLE

I have never seen this particular crash again.  Tentatively marking this as "cant repeat".