Bug #41485 mysqld got signal 11;
Submitted: 16 Dec 2008 2:13 Modified: 16 Jan 2009 4:34
Reporter: Brendan Pike Email Updates:
Status: No Feedback Impact on me:
Category:MySQL Server Severity:S2 (Serious)
Version:5.0.51a OS:Linux (Debian Linux 2.6.22-3-vserver-686 #1 SMP)
Assigned to: CPU Architecture:Any

[16 Dec 2008 2:13] Brendan Pike
I am experiencing occasional crashes, approximately once a week mysqld will crash and restart itself and after several of these the mysqld process will hang leaving no option than to kill -9 the process.  There doesn't appear to be any consistent reason for the crash, perhaps associated with heavy server loads at the time but I've never been present during a crash to know.

Please let me know what other information I can provide to be helpful.

The following is output from the syslog::

mysqld[4241]: 081216  3:54:29 - mysqld got signal 11;
mysqld[4241]: This could be because you hit a bug. It is also possible that this binary
mysqld[4241]: or one of the libraries it was linked against is corrupt, improperly built,
mysqld[4241]: or misconfigured. This error can also be caused by malfunctioning hardware.
mysqld[4241]: We will try our best to scrape up some info that will hopefully help diagnose
mysqld[4241]: the problem, but since we have already crashed, something is definitely wrong 
mysqld[4241]: and this may fail. 
mysqld[4241]: key_buffer_size=16777216
mysqld[4241]: read_buffer_size=131072
mysqld[4241]: max_used_connections=101
mysqld[4241]: max_connections=100
mysqld[4241]: threads_connected=8
mysqld[4241]: It is possible that mysqld could use up to 
mysqld[4241]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 233983 K
mysqld[4241]: bytes of memory
mysqld[4241]: Hope that's ok; if not, decrease some variables in the equation.
mysqld[4241]: thd=0xb0bf1558
mysqld[4241]: Attempting backtrace. You can use the following information to find out
mysqld[4241]: where mysqld died. If you see no messages after this, something went
mysqld[4241]: terribly wrong...
mysqld[4241]: Cannot determine thread, fp=0xaf2dcbf8, backtrace may not be correct.
mysqld[4241]: Stack range sanity check OK, backtrace follows:
mysqld[4241]: 0x81f3811
mysqld[4241]: 0x82efef3
mysqld[4241]: 0x82f2bbe
mysqld[4241]: 0x82f319b
mysqld[4241]: 0x82f3951
mysqld[4241]: 0x82074ae
mysqld[4241]: 0x820ddfe
mysqld[4241]: 0x8210783
mysqld[4241]: 0x821120a
mysqld[4241]: 0x8211b84
mysqld[4241]: 0xb7f494c0
mysqld[4241]: 0xb7d5b61e
mysqld[4241]: New value of fp=(nil) failed sanity check, terminating stack trace!
mysqld[4241]: Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
mysqld[4241]: stack trace is much more helpful in diagnosing the problem, so please do 
mysqld[4241]: resolve it
mysqld[4241]: Trying to get some variables.
mysqld[4241]: Some pointers may be invalid and cause the dump to abort...
mysqld[4241]: thd->query at 0xafbdcf70  is invalid pointer
mysqld[4241]: thd->thread_id=120804
mysqld[4241]: The manual page at http://www.mysql.com/doc/en/Crashing.html contains
mysqld[4241]: information that should help you find out what is causing the crash.
mysqld_safe[12144]: Number of processes running now: 0
mysqld_safe[12148]: restarted
Dmysqld[12151]: InnoDB: Log scan progressed past the checkpoint lsn 14 3920410572
Dmysqld[12151]: 081216  3:54:33  InnoDB: Database was not shut down normally!
Dmysqld[12151]: InnoDB: Starting crash recovery.
Dmysqld[12151]: InnoDB: Reading tablespace information from the .ibd files...
mysqld[12151]: InnoDB: Restoring possible half-written data pages from the doublewrite
mysqld[12151]: InnoDB: buffer...
mysqld[12151]: InnoDB: Doing recovery: scanned up to log sequence number 14 3920419010
mysqld[12151]: 081216  3:54:41  InnoDB: Starting an apply batch of log records to the database...
mysqld[12151]: InnoDB: Progress in percents: 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
mysqld[12151]: InnoDB: Apply batch completed
mysqld[12151]: InnoDB: Last MySQL binlog file position 0 13002192, file name /var/log/mysql/mysql-bin.000963
mysqld[12151]: 081216  3:54:42  InnoDB: Started; log sequence number 14 3920419010
mysqld[12151]: 081216  3:54:42 [Note] Recovering after a crash using /var/log/mysql/mysql-bin
mysqld[12151]: 081216  3:54:43 [Note] Starting crash recovery...
mysqld[12151]: 081216  3:54:43 [Note] Crash recovery finished.
mysqld[12151]: 081216  3:54:44 [Note] /usr/sbin/mysqld: ready for connections.
mysqld[12151]: Version: '5.0.51a-18-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)

Last entry in /var/log/mysql/mysql-bin.000963 ::
)~RFI^B^A^@^@^@�^A^@^@t|^@^@^@^A^@^@^@^@^@^K^@^@^Z^@^@^@@^@^@^A^@^@^@^@^@^@^@^@^F^Cstd^D^H^@^H^@^H^@kangaroo_is^@INSERT INTO eZSession_SessionSess
                                    ( ID, Created, LastAccessed, Hash1 )
                             VALUES ( '1756225',

How to repeat:
Unfortunately I do not know what is causing this crash and therefore can't repeat it at will.
[16 Dec 2008 4:30] Brendan Pike
Not sure if its just coincidence but I have just discovered that this error first during a automated mysqldump of all 40 databases.  The mysqldump completed the dump of the 4th database and never got any further, each subsequent sql mysqldump file was 0 bytes.  The time exactly matches the first "mysqld got signal 11;" syslog message of mysqld.  We have been doing backups like this for years but never experienced these problems until recently.
[16 Dec 2008 4:34] Valeriy Kravchuk
Thank you for a poblem report. Can you, please, send resolved stack trace. See http://dev.mysql.com/doc/refman/5.0/en/resolve-stack-dump.html for the details.
[17 Jan 2009 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".