Bug #32431 MySQLd crashes after double free or corruption error
Submitted: 16 Nov 2007 9:57 Modified: 20 Apr 2008 9:58
Reporter: Jari Ketola Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: General Severity:S2 (Serious)
Version:5.0.22, 5.0.45 OS:Linux (2.6.18-8.1.15.el5 x86_64)
Assigned to: CPU Architecture:Any
Tags: double free or corruption, glibc, signal 6, x86_64

[16 Nov 2007 9:57] Jari Ketola
Description:
MySQLd crashes unexpectedly at random intervals with the following error:

*** glibc detected *** /usr/libexec/mysqld: double free or corruption (!prev): 0x000000001104c1e0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x34d026e890]
/lib64/libc.so.6(cfree+0x8c)[0x34d0271fac]
/usr/libexec/mysqld(free_root+0x45)[0x808fb5]
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x220)[0x560690]
/usr/libexec/mysqld(_Z10do_commandP3THD+0x96)[0x561946]
/usr/libexec/mysqld(handle_one_connection+0x990)[0x5623f0]
/lib64/libpthread.so.0[0x34d0e061b5]
/lib64/libc.so.6(clone+0x6d)[0x34d02cd36d]
======= Memory map: ========
mysqld got signal 6;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=33554432
read_buffer_size=2093056
max_used_connections=112
max_connections=120
threads_connected=21
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 4210207 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

The server is running x86_64 Kernel 2.6.18-8.1.15.el5 (CentOS release 5).

# uname -a
Linux db2.afterdawn.net 2.6.18-8.1.15.el5 #1 SMP Mon Oct 22 08:32:28 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

# free -m
             total       used       free     shared    buffers     cached
Mem:          7987       7782        204          0        303       4728
-/+ buffers/cache:       2750       5236
Swap:         1992          0       1992

We are using the CentOS RPM for MySQL.

# rpm -qi mysql-server
Name        : mysql-server                 Relocations: (not relocatable)
Version     : 5.0.22                            Vendor: CentOS
Release     : 2.1.0.1                       Build Date: Thu 30 Aug 2007 02:52:08 PM UTC
Install Date: Wed 26 Sep 2007 06:58:11 AM UTC      Build Host: builder6
Group       : Applications/Databases        Source RPM: mysql-5.0.22-2.1.0.1.src.rpm
Size        : 23629978                         License: GPL
Signature   : DSA/SHA1, Fri 31 Aug 2007 12:36:32 AM UTC, Key ID a8a447dce8562897

The problem occurs on two of our database servers (both the master and slave), but is more prominent on the slave server which is running under more load (around 215 queries per second vs 85 qps, 1.5 load average vs. 0.25 load average).

At times the crash only causes the slave instance to stop while the server continues responding to queries. We have had to build a monitoring system to automatically shut down the database, repair MyISAM tables and bring the server back up in case of a failure.

It's also worth mentioning that the hardware on the server has been extensively tested (especially the memory), and no problems have been discovered. The server is running ECC memory and EDAC hasn't reported any parity errors on the system.

How to repeat:
Unknown.
[16 Nov 2007 10:31] Valeriy Kravchuk
Thank you for a problem report. Please, try to repeat with a newer version, 5.0.45, and inform about the results.
[27 Nov 2007 8:05] Jari Ketola
Running MySQL-server-community v5.0.45 now. I will update the bug after observing the server behavior for a week or so. Thanks.
[28 Dec 2007 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[8 Jan 2008 12:11] Jari Ketola
Now running

# mysqld --version
mysqld  Ver 5.0.45-community-log for unknown-linux-gnu on x86_64 (MySQL Community Edition (GPL))

After working fine for a while the server crashed again:

*** glibc detected *** /usr/sbin/mysqld: corrupted double-linked list: 0x000000001a276060 ***

080103 13:32:55 - mysqld got signal 6;

Still running kernel 2.6.18-8.1.15.el5 #1 SMP Mon Oct 22 08:32:28 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
[17 Feb 2008 16:55] Valeriy Kravchuk
Please, try to repeat with a newer version, 5.0.51a, and inform about the results.
[18 Mar 2008 0:02] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[20 Mar 2008 9:58] Susanne Ebrecht
Please, let us know what exactly you did to get this error.
Also we need to know how you installed MySQL and the version of your libc.
[20 Apr 2008 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".