MySQL Bugs: #49361: mysqld got signal 11,

Bug #49361	mysqld got signal 11,
Submitted:	2 Dec 2009 18:16	Modified:	29 May 2010 5:14
Reporter:	Kavita T	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	5.1.37	OS:	Linux (2.6.9-78.ELsmp)
Assigned to:		CPU Architecture:	Any
Tags:	mysqld_list_processes, Signal 11

Description:
091201 11:17:15 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1572864000
read_buffer_size=131072
max_used_connections=4
max_threads=1000
threads_connected=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2186414 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x155ad260
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...stack_bottom = 0x45089138 thread_stack 0x40000<path>/mysqld(my_print_stacktrace+0x26) [0x972c76]<PATH>/libexec64/mysqld(handle_segfault+0x253) [0x62f797]
/lib64/tls/libpthread.so.0 [0x36b590c5b0]/lib64/tls/libc.so.6(strlen+0x30) [0x36b4c70be0]
<path>/libexec64/mysqld(strdup_root+0x21) [0x95dec1]<PATH>/libexec64/mysqld(Query_arena::strdup(char const*)+0x21) [0x576c89]
<path>/libexec64/mysqld(mysqld_list_processes(THD*, char const*, bool)+0x5ac) [0x791fa2]
<path>/libexec64/mysqld(mysql_execute_command(THD*)+0x2d46) [0x642738]
<path>/libexec64/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x22a) [0x6491ba]
<PATH>/libexec64/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xa23) [0x63e2f1]
<PATH>/libexec64/mysqld(do_command(THD*)+0x1e8) [0x63d67a]
<PATH>/libexec64/mysqld(handle_one_connection+0x179) [0x63bb9b]
/lib64/tls/libpthread.so.0 [0x36b5906137]
/lib64/tls/libc.so.6(__clone+0x73) [0x36b4cc9883]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x155cdd40 = SHOW PROCESSLIST
thd->thread_id=86550
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

How to repeat:
cannot reproduce at will

Backtrace looks similar to bug: http://bugs.mysql.com/bug.php?id=49189?

Agree, however I have not come across a solution to avoid/mitigate this error.

Also note that bug http://bugs.mysql.com/bug.php?id=49189? refers to mysql version 5.1.32, whereas we're on 5.1.37

Here's a backtrace with symbols from a recent such crash:

#0  0x0000003a7ab098b7 in pthread_kill () from /lib64/tls/libpthread.so.0
(gdb) bt
#0  0x0000003a7ab098b7 in pthread_kill () from /lib64/tls/libpthread.so.0
#1  0x000000000062f94b in handle_segfault (sig=11) at mysqld.cc:2552
#2  <signal handler called>
#3  0x0000003a79e70be0 in strlen () from /lib64/tls/libc.so.6
#4  0x000000000095dec1 in strdup_root (root=0x15527650, str=0x0) at my_alloc.c:403
#5  0x0000000000576c89 in Query_arena::strdup (this=0x15524cf8, str=0x0) at sql_class.h:564
#6  0x0000000000791fa2 in mysqld_list_processes (thd=0x15524ce0, user=0x0, verbose=false) at sql_show.cc:1774
#7  0x0000000000642738 in mysql_execute_command (thd=0x15524ce0) at sql_parse.cc:3461
#8  0x00000000006491ba in mysql_parse (thd=0x15524ce0, inBuf=0x2cd8404fe0 "SHOW PROCESSLIST", length=16, found_semicolon=0x4514be98)
    at sql_parse.cc:6173
#9  0x000000000063e2f1 in dispatch_command (command=COM_QUERY, thd=0x15524ce0, packet=0x1548b781 "", packet_length=16) at sql_parse.cc:1307
#10 0x000000000063d67a in do_command (thd=0x15524ce0) at sql_parse.cc:945
#11 0x000000000063bb9b in handle_one_connection (arg=0x155550c0) at sql_connect.cc:1142
#12 0x0000003a7ab06137 in start_thread () from /lib64/tls/libpthread.so.0
#13 0x0000003a79ec9883 in clone () from /lib64/tls/libc.so.6

Tracking down the right source code so I can look at those line numbers.  A review of the latest source didn't show anything obvious, but I barely read C/C++, so that's not surprising!

So, if this is iterating over the process list, without being properly surrounded by a mutex, arbitrary garbage (including a null pointer) could happen.

There seems to be a mutex (LOCK_thread_count?) around this code, perhaps some other code is being naughty?  What thread-related code was added at, or shortly before, 5.1.32?

Another crash observed - 
091220 20:33:18 - mysqld got signal 11 ;
stack_bottom = 0x4514c138 thread_stack 0x40000
/home/y/libexec64/mysqld(my_print_stacktrace+0x26) [0x972c76]
/home/y/libexec64/mysqld(handle_segfault+0x253) [0x62f797]
/lib64/tls/libpthread.so.0 [0x3a7ab0c5b0]
/lib64/tls/libc.so.6(strlen+0x30) [0x3a79e70be0]
/home/y/libexec64/mysqld(strdup_root+0x21) [0x95dec1]
/home/y/libexec64/mysqld(Query_arena::strdup(char const*)+0x21) [0x576c89]
/home/y/libexec64/mysqld(mysqld_list_processes(THD*, char const*, bool)+0x5ac) [0x791fa2]
/home/y/libexec64/mysqld(mysql_execute_command(THD*)+0x2d46) [0x642738]
/home/y/libexec64/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x22a) [0x6491ba]
/home/y/libexec64/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xa23) [0x63e2f1]
/home/y/libexec64/mysqld(do_command(THD*)+0x1e8) [0x63d67a]
/home/y/libexec64/mysqld(handle_one_connection+0x179) [0x63bb9b]
/lib64/tls/libpthread.so.0 [0x3a7ab06137]
/lib64/tls/libc.so.6(__clone+0x73) [0x3a79ec9883]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x15616550 = SHOW PROCESSLIST
thd->thread_id=289596
thd->killed=NOT_KILLED

core_file_dump

Attachment: core_generate_extra.log (text/x-log), 14.25 KiB.

backtrace from a recent crash

Attachment: core_generate.log (text/x-log), 2.87 KiB.

(gdb) bt
#0  0x0000003a7ab098b7 in pthread_kill () from /lib64/tls/libpthread.so.0
#1  0x000000000062f94b in handle_segfault (sig=11) at mysqld.cc:2552
#2  <signal handler called>
#3  0x0000003a79e70be0 in strlen () from /lib64/tls/libc.so.6
#4  0x000000000095dec1 in strdup_root (root=0x15642c40, str=0x0) at my_alloc.c:403
#5  0x0000000000576c89 in Query_arena::strdup (this=0x156402e8, str=0x0) at sql_class.h:564
#6  0x0000000000791fa2 in mysqld_list_processes (thd=0x156402d0, user=0x0, verbose=false) at sql_show.cc:1774
#7  0x0000000000642738 in mysql_execute_command (thd=0x156402d0) at sql_parse.cc:3461
#8  0x00000000006491ba in mysql_parse (thd=0x156402d0, inBuf=0x1564dd00 "SHOW PROCESSLIST", length=16, found_semicolon=0x4518ce98) at sql_parse.cc:6173
#9  0x000000000063e2f1 in dispatch_command (command=COM_QUERY, thd=0x156402d0, packet=0x15642c81 "", packet_length=16) at sql_parse.cc:1307
#10 0x000000000063d67a in do_command (thd=0x156402d0) at sql_parse.cc:945
#11 0x000000000063bb9b in handle_one_connection (arg=0x156a6d70) at sql_connect.cc:1142
#12 0x0000003a7ab06137 in start_thread () from /lib64/tls/libpthread.so.0
#13 0x0000003a79ec9883 in clone () from /lib64/tls/libc.so.6
(gdb)

Please, check if this is repeatable with a newer version, 5.1.41.

Thanks Valeriy, part of the problem lies in not being able to reproduce this error at will on any 5.1.?? versions. This error appears to be occurring randomly in our production environment. we tried creating a synthetic crash (signal 11) with a number of concurrent sessions issuing 'show processlist' on the database, but havent been able to induce a crash. Unless we can build a test case, it is unknown at this point if an upgrade will fix this issue.

Thanks Valeriy, part of the problem lies in not being able to reproduce this error at will on any 5.1.?? versions. This error appears to be occurring randomly in our production environment. we tried creating a synthetic crash (signal 11) with a number of concurrent sessions issuing 'show processlist' on the database, but havent been able to induce a crash. Unless we build/prove a test case, it is unknown at this point if an upgrade will fix this issue.

we believe this is due to 5.1.37 build --with-debug, can anyone comment/confirm this ?

Thank you for the feedback.

I can not repeat described behavior. Could you please send us configuration file also? Please also consider to test with verison 5.1.47 in your environment in case if bug uccasionally fixed.

Fixed by upgrading to 5.1.41

Thank you for the feedback.

Closed as "Can't repeat"