MySQL Bugs: #34443: *** glibc detected *** corrupted double-linked list: *** mysqld got signal 6;

Bug #34443	* glibc detected * corrupted double-linked list: *** mysqld got signal 6;
Submitted:	9 Feb 2008 9:04	Modified:	31 Dec 2010 19:06
Reporter:	kris dba	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	4.1.22, 5.0.90, 5.1.46	OS:	Linux
Assigned to:		CPU Architecture:	Any
Tags:	corrupted double-linked list, glibc, signal 6, x86_64

Description:
Hi,
mysqld crashed giving the following alerts in error log.
The server is acting as Replication Master.
*** glibc detected *** corrupted double-linked list: ***
mysqld got signal 6;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went terribly wrong
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread,backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
New value of fp=0x30035b0 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
Number of processes running now: 0

How to repeat:
Unknown

Thank you for a problem report. Please, check if any similar error ever happens with a newer version, 4.1.22 or 4.1.23.

Thank you for the feedback.

Please try to provide repeatable test case.

Hi,
The crash occured on system which is acting as mysql slave.
sytem got crashed while altering a merge table to add  new myisam table to the list.
readlock was applied on the table to stop the updates from the master.

possible updates from master would have caused the crash..

Thank you for the feedback. How Sveta asked you before could you please provide a repeatable test case. Thanks in advance.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

MySql server error

Attachment: mysql.err (application/octet-stream, text), 14.11 KiB.

We recently upgraded from 5.0 to 5.1.46 running Gentoo 2.6.31-hardened-r10.  We never used to get this error, but are getting this every few days now.  Please help!

Error Log for the above post

Attachment: la-dbb1 mysql crash 20100617.rtf (application/rtf, text), 22.09 KiB.

Thank you for the feedback.

2 error logs show crashes were in different place of code. Please check physical RAM to be sure it does not have error. If RAM is OK, please, check if SELECT query from the error log which you sent first crashes server every time.

MySql server error log

Attachment: mysql.err (application/octet-stream, text), 16.12 KiB.

Hello

same problem appeared yesterday; unlike the other day, it happened under decreasing load after a full day performing fine with far bigger loads.

Thanks for your help

@Sveta Smirnova 

RAM does not report any error, and the corresponding SQL query is run thousands time a day.

May it be a corrupted storage item (we use both RAID1 SSD & RAID1 disk) ?

Thx

Only one of our tables is still on MyIsam (and reported crashed) after MySql restart, we'll convert it to InnoDb tonight to see if there is any stability improvement.

Thank you for the feedback.

Yes, checking disks can be helpful too.

If crashing table is MyISAM table running CHECK TABLE can make sense. Please inform us if there is any error.

Hello,

the check of the table did not show anything wrong; table was repaired & converted to InnoDb like all the others.

We need to plan downtime for the disk check, probably this evening.

Thanks

We've downgraded to 5.0.90, and I'm still getting this error.  This always happens under load.  We're loading files on the magnitude of 100 mil records.

ERROR MESSAGE in /var/log/mysql/mysqld.err:
100701 13:20:57 [Note] /usr/sbin/mysqld: Normal shutdown

100701 13:20:57 [Note] Event Scheduler: Purging the queue. 0 events
100701 13:20:59  InnoDB: Starting shutdown...
100701 13:21:11  InnoDB: Shutdown completed; log sequence number 194 882449437
100701 13:21:11 [Note] /usr/sbin/mysqld: Shutdown complete

InnoDB: The InnoDB memory heap is disabled
InnoDB: use atomic builtins.
100701 13:21:21  InnoDB: Started; log sequence number 194 882449437
100701 13:21:24 [ERROR] log *** glibc detected *** /usr/sbin/mysqld: corrupted double-linked list: 0x00006fcd0c077670 *** listed in the index, but failed to stat
100701 13:21:24 [ERROR] Error counting relay log space
100701 13:21:24 [ERROR] Failed to initialize the master info structure
100701 13:21:24 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.90-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Gentoo Linux mysql-5.0.90-r2
100701 13:22:25 [ERROR] log *** glibc detected *** /usr/sbin/mysqld: corrupted double-linked list: 0x00006fcd0c077670 *** listed in the index, but failed to stat
100701 13:22:25 [ERROR] Error counting relay log space

SLAVE STATUS:
*************************** 1. row ***************************
             Slave_IO_State:
                Master_Host: 192.168.201.102
                Master_User: repl
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File:
        Read_Master_Log_Pos: 4
             Relay_Log_File: la-dba1-relay-bin.000267
              Relay_Log_Pos: 4
      Relay_Master_Log_File:
           Slave_IO_Running: No
          Slave_SQL_Running: No
            Replicate_Do_DB:
        Replicate_Ignore_DB:
         Replicate_Do_Table:
     Replicate_Ignore_Table:
    Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
                 Last_Errno: 0
                 Last_Error:
               Skip_Counter: 0
        Exec_Master_Log_Pos: 0
            Relay_Log_Space: 512
            Until_Condition: None
             Until_Log_File:
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File:
         Master_SSL_CA_Path:
            Master_SSL_Cert:
          Master_SSL_Cipher:
             Master_SSL_Key:
      Seconds_Behind_Master: NULL
1 row in set (0.00 sec)

Lionel,

thank you for the feedback. As you sent no test case I assume workaround: setting innodb_max_dirty_pages_pct  to 10 works and problem is not repeatable any more. So I close the report as "Can't repeat".

Agreed. Please close.