MySQL Bugs: #49189: server crashes randomly with signal 11

Bug #49189	server crashes randomly with signal 11
Submitted:	29 Nov 2009 23:13	Modified:	25 Jan 2010 17:38
Reporter:	Nirmal Thangaraj	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: General	Severity:	S1 (Critical)
Version:	5.1.32.1	OS:	Linux ( 2.6.9-67.ELsmp )
Assigned to:		CPU Architecture:	Any
Tags:	crash, Signal 11

Description:
Database crashes randomly with a Signal 11 message. I am pasting two latest entries in the error log file, below.

Other Signal 11 bugs ( Bug#48239 and Bug#34236 ) for 5.1 seem related to SSL, which is not the case here.

------------------------------
091127 13:52:49 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1001
max_threads=1000
threads_connected=383
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 781368 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x99607a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4fd72138 thread_stack 0x40000
/home/y/libexec64/mysqld(my_print_stacktrace+0x26) [0x9d0e16]
/home/y/libexec64/mysqld(handle_segfault+0x253) [0x6342a5]
/lib64/tls/libpthread.so.0 [0x37a370c5b0]
/lib64/tls/libc.so.6(strlen+0x30) [0x37a2e70a30]
/home/y/libexec64/mysqld(strdup_root+0x33) [0x9beaa3]
/home/y/libexec64/mysqld(Query_arena::strdup(char const*)+0x21) [0x58394b]
/home/y/libexec64/mysqld(mysqld_list_processes(THD*, char const*, bool)+0x535) [0x76c23d]
/home/y/libexec64/mysqld(mysql_execute_command(THD*)+0x2940) [0x644dbe]
/home/y/libexec64/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x1da) [0x64af06]
/home/y/libexec64/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x95f) [0x640fa5]
/home/y/libexec64/mysqld(do_command(THD*)+0x13c) [0x6404be]
/home/y/libexec64/mysqld(handle_one_connection+0x158) [0x63ed08]
/lib64/tls/libpthread.so.0 [0x37a3706137]
/lib64/tls/libc.so.6(__clone+0x73) [0x37a2ec7533]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2c23e1c330 is an invalid pointer
thd->thread_id=5569680
thd->killed=NOT_KILLED
--------------

091128  7:42:52 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1001
max_threads=1000
threads_connected=401
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 781368 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x2b554aa400
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4b001138 thread_stack 0x40000
/home/y/libexec64/mysqld(my_print_stacktrace+0x26) [0x9d0e16]
/home/y/libexec64/mysqld(handle_segfault+0x253) [0x6342a5]
/lib64/tls/libpthread.so.0 [0x37a370c5b0]
/lib64/tls/libc.so.6(strlen+0x30) [0x37a2e70a30]
/home/y/libexec64/mysqld(strdup_root+0x33) [0x9beaa3]
/home/y/libexec64/mysqld(Query_arena::strdup(char const*)+0x21) [0x58394b]
/home/y/libexec64/mysqld(mysqld_list_processes(THD*, char const*, bool)+0x535) [0x76c23d]
/home/y/libexec64/mysqld(mysql_execute_command(THD*)+0x2940) [0x644dbe]
/home/y/libexec64/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x1da) [0x64af06]
/home/y/libexec64/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x95f) [0x640fa5]
/home/y/libexec64/mysqld(do_command(THD*)+0x13c) [0x6404be]
/home/y/libexec64/mysqld(handle_one_connection+0x158) [0x63ed08]
/lib64/tls/libpthread.so.0 [0x37a3706137]
/lib64/tls/libc.so.6(__clone+0x73) [0x37a2ec7533]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2b55f5a4b0 is an invalid pointer
thd->thread_id=177276
thd->killed=NOT_KILLED
---------------------

Thanks
- Nirmal

How to repeat:
There is no single query which causes DB to crash. DB has been running fine with thousands of queries for days, and suddenly crashes with signal 11, with no change in query or load.

Suggested fix:
unknown

Thank you for the problem report. Please, send your my.cnf file content and the results of:

uname -a
free

Linux commands.

The results of SHOW GLOBAL STATUS statement would be also useful.

$uname -a
Linux <hostname> 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:56:44 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

$free
             total       used       free     shared    buffers     cached
Mem:      16406444   16344660      61784          0     146240   11519636
-/+ buffers/cache:    4678784   11727660
Swap:      8388600        208    8388392

All the tables in the DB are InnoDB. 

Thanks
- Nirmal

Thank you for the feedback.

You experience crashes with same trace. Could you please try to catch the query which causes the problem? You can use general query log for it (this would be one of latest queries before crash). Alternatively you can use proxy with custom scripting or use debug version of server, then upload core file.

Started general query log. will update once I get some data.

Thanks
- Nirmal

Thank you for the update.

We will wait when you have more results.

See bug: http://bugs.mysql.com/bug.php?id=49361.

Well, the DB crashed again 

--------------------------------------------------------------------
091216 21:28:37 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1116
max_threads=3000
threads_connected=235
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2081962 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x2bbac5a450
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4f183138 thread_stack 0x40000
/home/y/libexec64/mysqld(my_print_stacktrace+0x26) [0x9d0e16]
/home/y/libexec64/mysqld(handle_segfault+0x253) [0x6342a5]
/lib64/tls/libpthread.so.0 [0x306be0c5b0]
/lib64/tls/libc.so.6(strlen+0x30) [0x306b570a30]
/home/y/libexec64/mysqld(strdup_root+0x33) [0x9beaa3]
/home/y/libexec64/mysqld(Query_arena::strdup(char const*)+0x21) [0x58394b]
/home/y/libexec64/mysqld(mysqld_list_processes(THD*, char const*, bool)+0x535) [0x76c23d]
/home/y/libexec64/mysqld(mysql_execute_command(THD*)+0x2940) [0x644dbe]
/home/y/libexec64/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x1da) [0x64af06]
/home/y/libexec64/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x95f) [0x640fa5]
/home/y/libexec64/mysqld(do_command(THD*)+0x13c) [0x6404be]
/home/y/libexec64/mysqld(handle_one_connection+0x158) [0x63ed08]
/lib64/tls/libpthread.so.0 [0x306be06137]
/lib64/tls/libc.so.6(__clone+0x73) [0x306b5c7533]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2c0c8870b0 is an invalid pointer
thd->thread_id=1589385
thd->killed=NOT_KILLED
--------------------------------------------------------------------

I am pasting the commands in the general log just before the crash

--------------------------------------------------------------------
091216 21:28:371584749 Query    CALL <stored-proc>
                66733 Query     BEGIN
                66733 Query     UPDATE <stmt>
                66733 Query     COMMIT /* implicit, from Xid_log_event */
                1510734 Query   CALL <stored-proc>
                66733 Query     BEGIN
                66733 Query     UPDATE <stmt>
                66733 Query     COMMIT /* implicit, from Xid_log_event */
                1545276 Query   CALL <stored-proc>
                1584749 Query   CALL <stored-proc>
                1584485 Query   CALL <stored-proc>
                1589385 Connect adminuser@localhost on
                1589385 Query   SHOW PROCESSLIST
                1589386 Connect adminuser@localhost on
Tcp port: 3306  Unix socket: /tmp/mysql.sock
Time                 Id Command    Argument
091216 21:44:14    1 Connect    adminuser@localhost on
                    1 Query     SHOW PROCESSLIST

--------------------------------------------------------------------

Can 'SHOW PROCESSLIST' be the cause of this crash?

- Nirmal

Please, check if this is repeatable with a newer version, 5.1.41.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".