Bug #19154 MySQLD Core, glibc detected double free or corruption (fasttop)
Submitted: 17 Apr 2006 23:47 Modified: 10 May 2006 16:07
Reporter: Jonathan Miller Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:5.1.10 OS:Linux (Linux 32 Bit OS)
Assigned to: Jonas Oreland CPU Architecture:Any

[17 Apr 2006 23:47] Jonathan Miller
Description:
Running load / stress testing today I recieved a  glibc detected message as the mysql cored.

I have 2 regular and 2 stored procedure TPC-B test running against mixed memory and disk data, 1 - DBT2(TPC-C) and  4 asyncndbtest running against a 4 DN, 2 replica cluster.

On the ndb_mgmd data node I was logged into MySQL client and had no processes running against the MySQLD on that host.

When I swtiched back from looking at other items i found:

mysql> *** glibc detected *** /home/ndbdev/jmiller/builds/libexec/mysqld: double free or corruption (fasttop): 0x09e18658 ***
======= Backtrace: =========
/lib/libc.so.6[0x8ee124]
/lib/libc.so.6(__libc_free+0x77)[0x8ee65f]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZdaPv+0x17)[0x8420fd3]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN17EventBufData_list10add_gci_opENS_6Gci_opEb+0x138)[0x83625f4]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN17EventBufData_list11append_dataEP12EventBufData+0x51)[0x83684a7]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN14NdbEventBuffer11insertDataLEP21NdbEventOperationImplPK12SubTableDataP16LinearSectionPtr+0x1a5)[0x8366d25]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN3Ndb20handleReceivedSignalEP12NdbApiSignalP16LinearSectionPtr+0xfcf)[0x83455bd]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN3Ndb14executeMessageEPvP12NdbApiSignalP16LinearSectionPtr+0x2a)[0x8345ba6]
/home/ndbdev/jmiller/builds/libexec/mysqld(_Z7executePvP12SignalHeaderhPjP16LinearSectionPtr+0xc9)[0x836cc8d]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN19TransporterRegistry6unpackEPjjt7IOState+0x28c)[0x83c40c8]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN19TransporterRegistry14performReceiveEv+0x144)[0x839307c]
/home/ndbdev/jmiller/builds/libexec/mysqld(_ZN17TransporterFacade17threadMainReceiveEv+0xe0)[0x836b620]
/home/ndbdev/jmiller/builds/libexec/mysqld(runReceiveResponse_C+0x1f)[0x836b6d1]
/home/ndbdev/jmiller/builds/libexec/mysqld[0x83b005c]
/lib/libpthread.so.0[0x9fdb80]
/lib/libc.so.6(__clone+0x5e)[0x9559ce]
 

How to repeat:
Not sure.
[17 Apr 2006 23:50] Jonathan Miller
060417 19:32:32 [ERROR] NDB tablespace: distributing ./ts1 timed out. Ignoring...
060417 19:32:43 [ERROR] NDB create table: distributing ./TESTER_2/t1 timed out. Ignoring...
060417 19:33:01 [ERROR] NDB drop table: distributing ./TESTER_2/t1 timed out. Ignoring...
060417 19:33:12 [ERROR] NDB drop db: distributing ./TESTER_2/ timed out. Ignoring...
060417 19:33:23 [ERROR] NDB tablespace: distributing ./ts1 timed out. Ignoring...
060417 19:33:34 [ERROR] NDB tablespace: distributing ./ts1 timed out. Ignoring...
060417 19:33:45 [ERROR] NDB logfile group: distributing ./lg1 timed out. Ignoring...
060417 21:34:30 [ERROR] NDB create db: distributing ./TPCB/ timed out. Ignoring...
060417 21:36:40 [ERROR] NDB logfile group: distributing ./lg1 timed out. Ignoring...
060417 21:40:46 [ERROR] NDB tablespace: distributing ./ts1 timed out. Ignoring...
060417 21:44:54 [ERROR] NDB tablespace: distributing ./ts1 timed out. Ignoring...
060417 21:45:06 [ERROR] NDB create table: distributing ./TPCB/account timed out. Ignoring...
060417 21:45:18 [ERROR] NDB create table: distributing ./TPCB/branch timed out. Ignoring...
060417 21:45:30 [ERROR] NDB create table: distributing ./TPCB/teller timed out. Ignoring...
060417 21:45:43 [ERROR] NDB create table: distributing ./TPCB/history timed out. Ignoring...
060417 21:45:55 [ERROR] NDB create table: distributing ./TPCB/trans timed out. Ignoring...
[17 Apr 2006 23:51] Jonathan Miller
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=3
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x89d5650
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb7faef44, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81d0974
0xe29420
(nil)
0x9fdb80
0x9559ce
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
060418 01:07:46  mysqld restarted
/home/ndbdev/jmiller/builds/libexec/mysqld: Table 'general_log' is marked as crashed and should be repaired
/home/ndbdev/jmiller/builds/libexec/mysqld: Table 'slow_log' is marked as crashed and should be repaired
060418  1:07:51 [Note] Starting MySQL Cluster Binlog Thread
060418  1:07:51 [Note] /home/ndbdev/jmiller/builds/libexec/mysqld: ready for connections.
Version: '5.1.10-beta-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x97f59d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb1f6bf44, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81d0974
0xaff420
(nil)
0x9fdb80
0x9559ce
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
[17 Apr 2006 23:51] Jonathan Miller
Number of processes running now: 0
060418 01:08:40  mysqld restarted
/home/ndbdev/jmiller/builds/libexec/mysqld: Table 'general_log' is marked as crashed and should be repaired
/home/ndbdev/jmiller/builds/libexec/mysqld: Table 'slow_log' is marked as crashed and should be repaired
060418  1:08:45 [Note] Starting MySQL Cluster Binlog Thread
060418  1:08:45 [Note] /home/ndbdev/jmiller/builds/libexec/mysqld: ready for connections.
Version: '5.1.10-beta-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0xa22d9d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb1f33f44, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81d0974
0x8b3420
(nil)
0x9fdb80
0x1db9ce
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
060418 01:10:00  mysqld restarted
/home/ndbdev/jmiller/builds/libexec/mysqld: Table 'general_log' is marked as crashed and should be repaired
/home/ndbdev/jmiller/builds/libexec/mysqld: Table 'slow_log' is marked as crashed and should be repaired
060418  1:10:16 [Note] Starting MySQL Cluster Binlog Thread
060418  1:10:16 [Note] /home/ndbdev/jmiller/builds/libexec/mysqld: ready for connections.
Version: '5.1.10-beta-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x8f0b9d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb1f55f44, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81d0974
0xb85420
(nil)
0x9fdb80
0x9559ce
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
[17 Apr 2006 23:52] Jonathan Miller
Number of processes running now: 0
060418 01:11:55  mysqld restarted
/home/ndbdev/jmiller/builds/libexec/mysqld: Table 'general_log' is marked as crashed and should be repaired
/home/ndbdev/jmiller/builds/libexec/mysqld: Table 'slow_log' is marked as crashed and should be repaired
060418  1:12:06 [Note] Starting MySQL Cluster Binlog Thread
060418  1:12:06 [Note] /home/ndbdev/jmiller/builds/libexec/mysqld: ready for connections.
Version: '5.1.10-beta-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
mysqld got signal 6;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb1f4f36c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81d0974
0x6b9420
(nil)
0x8e822a
0x8ee124
0x8ee65f
0x8420fd3
0x83625f4
0x83684a7
0x8366d25
0x83455bd
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
[17 Apr 2006 23:52] Jonathan Miller
resolve_stack_dump -s /tmp/mysqld.sym -n stack.trace
0x81d0974 handle_segfault + 438
0x6b9420 (?)
(nil)
0x8e822a (?)
0x8ee124 (?)
0x8ee65f (?)
0x8420fd3 _ZdaPv + 23
0x83625f4 _ZN17EventBufData_list10add_gci_opENS_6Gci_opEb + 312
0x83684a7 _ZN17EventBufData_list11append_dataEP12EventBufData + 81
0x8366d25 _ZN14NdbEventBuffer11insertDataLEP21NdbEventOperationImplPK12SubTableDataP16LinearSectionPtr + 421
0x83455bd _ZN3Ndb20handleReceivedSignalEP12NdbApiSignalP16LinearSectionPtr + 4047
[21 Apr 2006 13:12] Tomas Ulin
several memory corrupting bugs have been fixed that are the likely cause of this, retesting is needed when those are merged into the main tree
[10 May 2006 7:19] Tomas Ulin
for documentation, fixed in 5.1.10
[10 May 2006 16:07] Paul DuBois
Noted in 5.1.10 changelog.

Eliminated some memory corruption problems that resultsd in
<literal>double free or corruption</literal> errors and a
server crash.