MySQL Bugs: #31834: Sporadic server crash on shutdown

Bug #31834	Sporadic server crash on shutdown
Submitted:	25 Oct 2007 7:27	Modified:	2 Aug 2008 16:24
Reporter:	Jan Lindström	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	5.1.22-RC	OS:	Linux
Assigned to:		CPU Architecture:	Any

Description:
I have problems with normal shutdown on mysql 5.1.22-RC. Here is the error log:

071025  9:44:33 [Note] /home/jan/mysql-5.1.22-rc/sql/mysqld: Normal shutdown

071025  9:44:33 [Note] Event Scheduler: Purging the queue. 0 events
solidDB Storage Engine for MySQL 5.1.22-rc - 06.01.0080 DEBUG   (Linux 2.6.20 ix86 MT)
This is a solidDB Storage Engine for MySQL 5.1
Copyright (C) Solid Information Technology Ltd 1993-2007
Creating a new database, please wait ...
Using log write mode 2
Merge started, 1431 keys to remove
Database started.
solidDB Storage Engine for MySQL is shut down
Total of 147 allocated pointers.
071025  9:44:37 [Note] /home/jan/mysql-5.1.22-rc/sql/mysqld: Shutdown complete

071025  9:44:37 - mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=2408550287
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2410938 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xa91f5028, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x826824d
0xb6b56f89
0xb7f42329
0xb7e4657e
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
and follow instructions on how to resolve the stack trace.
Resolved stack trace is much more helpful in diagnosing the
problem, so please do resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

jan@midearth:~/mysql-5.1.22-rc/sql$ ../extra/resolve_stack_dump ./mysqld_numeric 
0x826824d
0xb6b56f89
0xb7f42329
0xb7e4657e0x826824d handle_segfault + 647
0xb6b56f89 _end + -1372708207
0xb7f42329 _end + -1351821775
 
With ddd I managed to find that this happens on void clean_up(bool print_message) and the stack is:

Warning:
Cannot insert breakpoint -40.
Error accessing memory address 0x48ff82e8: Input/output error.

(gdb) where
#0  0xb7db6be0 in siglongjmp () from /lib/tls/i686/cmov/libc.so.6
#1  0xb7f5eb4c in unwind_stop () from /lib/tls/i686/cmov/libpthread.so.0
#2  0xb7f7dc7b in ?? () from /lib/libgcc_s.so.1
#3  0xb7f7dfbd in _Unwind_ForcedUnwind () from /lib/libgcc_s.so.1
#4  0xb7f60f86 in _Unwind_ForcedUnwind () from /lib/tls/i686/cmov/libpthread.so.0
#5  0xb7f5ea61 in __pthread_unwind () from /lib/tls/i686/cmov/libpthread.so.0
#6  0xb7f59410 in pthread_exit () from /lib/tls/i686/cmov/libpthread.so.0
#7  0x08267da0 in signal_hand (arg=0x0) at mysqld.cc:2481
#8  0xb7f5831b in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#9  0xb7e5c57e in clone () from /lib/tls/i686/cmov/libc.so.6

How to repeat:
killall mysqld

Thank you for a problem report. Please, send the results of:

getconf GNU_LIBPTHREAD_VERSION
getconf GNU_LIBC_VERSION
uname -a
free

Have you seen similar problem with previous versions of 5.1.x?

jan@midearth:~$ getconf GNU_LIBPTHREAD_VERSION
NPTL 2.5
jan@midearth:~$ getconf GNU_LIBC_VERSION
glibc 2.5
jan@midearth:~$ uname -a
Linux midearth 2.6.20-16-386 #2 Sun Sep 23 19:47:10 UTC 2007 i686 GNU/Linux
jan@midearth:~$ free
             total       used       free     shared    buffers     cached
Mem:       2075224    2024892      50332          0     465432     332928
-/+ buffers/cache:    1226532     848692
Swap:      9542528      34548    9507980

I have not tested older versions of 5.1 but similar problem is in 5.1.23-bk. I have seen this crash only when I use dynamic storage engine plugins.

Valgrind output

Attachment: valgrind.out (application/octet-stream, text), 20.79 KiB.

Please, try to repeat with a newer version, 5.1.25-rc, and inform about the results.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".