Bug #64254 Mysql 5.5.19 hit a bug
Submitted: 7 Feb 2012 20:56 Modified: 20 Apr 2012 16:39
Reporter: Flavian C Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Replication Severity:S1 (Critical)
Version:5.5.19 OS:Any
Assigned to: CPU Architecture:Any

[7 Feb 2012 20:56] Flavian C
Description:
Im running on SLES 11 SP1 x64
Mysql 5.5.19 x64 SLES11 RPM Build

How to repeat:
new mysql slave was pulling data from the master and after few minutes it gave me

*** glibc detected *** /usr/sbin/mysqld: double free or corruption (!prev): 0x00007f07786e5960 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75018)[0x7f08595d3018]
/lib64/libc.so.6(cfree+0x6c)[0x7f08595d7f6c]
/usr/sbin/mysqld(_ZN15Query_log_eventD0Ev+0x28)[0x73e558]
/usr/sbin/mysqld[0x53523c]
/usr/sbin/mysqld(handle_slave_sql+0xada)[0x53777a]
/lib64/libpthread.so.0(+0x75f0)[0x7f085a3635f0]
/lib64/libc.so.6(clone+0x6d)[0x7f085963284d]
======= Memory map: ========
00400000-00d1f000 r-xp 00000000 08:02 24477974                           /usr/sbin/mysqld
00f1f000-00f92000 r--p 0091f000 08:02 24477974                           /usr/sbin/mysqld
00f92000-01035000 rw-p 00992000 08:02 24477974                           /usr/sbin/mysqld
01035000-07ceb000 rw-p 00000000 00:00 0                                  [heap]
7f0778000000-7f077a62e000 rw-p 00000000 00:00 0
7f077a62e000-7f077c000000 ---p 00000000 00:00 0
7f077d02e000-7f077d02f000 ---p 00000000 00:00 0
7f077d02f000-7f077d05f000 rw-p 00000000 00:00 0
7f077d05f000-7f077d060000 ---p 00000000 00:00 0
7f077d060000-7f077d090000 rw-p 00000000 00:00 0
7f077d090000-7f077d091000 ---p 00000000 00:00 0
7f077d091000-7f077d0c1000 rw-p 00000000 00:00 0
7f077d0c1000-7f077d0c2000 ---p 00000000 00:00 0
7f077d0c2000-7f077d0f2000 rw-p 00000000 00:00 0
7f077d0f2000-7f077d0f3000 ---p 00000000 00:00 0
7f077d0f3000-7f077d8f3000 rw-p 00000000 00:00 0
7f077d8f3000-7f077d8f4000 ---p 00000000 00:00 0
7f077d8f4000-7f077e0f4000 rw-p 00000000 00:00 0
7f077e0f4000-7f077e0f5000 ---p 00000000 00:00 0
7f077e0f5000-7f077e8f5000 rw-p 00000000 00:00 0
7f077e8f5000-7f077e8f6000 ---p 00000000 00:00 0
7f077e8f6000-7f077ff26000 rw-p 00000000 00:00 0
7f077ff26000-7f077ff27000 ---p 00000000 00:00 0
7f077ff27000-7f0780727000 rw-p 00000000 00:00 0
7f0780727000-7f0780728000 ---p 00000000 00:00 0
7f0780728000-7f0780f28000 rw-p 00000000 00:00 0
7f0780f28000-7f0780f29000 ---p 00000000 00:00 0
7f0780f29000-7f0781729000 rw-p 00000000 00:00 0
7f0781729000-7f078172a000 ---p 00000000 00:00 0
7f078172a000-7f0781f2a000 rw-p 00000000 00:00 0
7f0781f2a000-7f0781f2b000 ---p 00000000 00:00 0
7f0781f2b000-7f078272b000 rw-p 00000000 00:00 0
7f078272b000-7f078272c000 ---p 00000000 00:00 0
7f078272c000-7f0782f2c000 rw-p 00000000 00:00 0
7f0782f2c000-7f0782f2d000 ---p 00000000 00:00 0
7f0782f2d000-7f078372d000 rw-p 00000000 00:00 0
7f078372d000-7f078372e000 ---p 00000000 00:00 0
7f078372e000-7f0783f2e000 rw-p 00000000 00:00 0
7f0783f2e000-7f0783f2f000 ---p 00000000 00:00 0
7f0783f2f000-7f078472f000 rw-p 00000000 00:00 0
7f078472f000-7f0784730000 ---p 00000000 00:00 0
7f0784730000-7f0784f30000 rw-p 00000000 00:00 0
7f0784f30000-7f0784f31000 ---p 00000000 00:00 0
7f0784f31000-7f0785731000 rw-p 00000000 00:00 0
7f0785731000-7f0785732000 ---p 00000000 00:00 0
7f0785732000-7f0785f32000 rw-p 00000000 00:00 0
7f0785f32000-7f0785f33000 ---p 00000000 00:00 0
7f0785f33000-7f0786733000 rw-p 00000000 00:00 0
7f0786733000-7f0786734000 ---p 00000000 00:00 0
7f0786734000-7f0786f34000 rw-p 00000000 00:00 0
7f0786f34000-7f0786f35000 ---p 00000000 00:00 0
7f0786f35000-7f0787735000 rw-p 00000000 00:00 0
7f0787735000-7f0787736000 ---p 00000000 00:00 0
7f0787736000-7f0787f36000 rw-p 00000000 00:00 0
7f0787f36000-7f0787f37000 ---p 00000000 00:00 0
7f0787f37000-7f0858b46000 rw-p 00000000 00:00 0
7f0858b46000-7f0858b5c000 r-xp 00000000 08:02 23896184                   /lib64/libgcc_s.so.1
7f0858b5c000-7f0858d5b000 ---p 00016000 08:02 23896184                   /lib64/libgcc_s.so.1
7f0858d5b000-7f0858d5c000 r--p 00015000 08:02 23896184                   /lib64/libgcc_s.so.1
7f0858d5c000-7f0858d5d000 rw-p 00016000 08:02 23896184                   /lib64/libgcc_s.so.1
7f0858d5d000-7f0858d5e000 ---p 00000000 00:00 0
7f0858d5e000-7f085955e000 rw-p 00000000 00:00 0
7f085955e000-7f08596b2000 r-xp 00000000 08:02 23896073                   /lib64/libc-2.11.1.so
7f08596b2000-7f08598b2000 ---p 00154000 08:02 23896073                   /lib64/libc-2.11.1.so
7f08598b2000-7f08598b6000 r--p 00154000 08:02 23896073                   /lib64/libc-2.11.1.so
7f08598b6000-7f08598b7000 rw-p 00158000 08:02 23896073                   /lib64/libc-2.11.1.so
7f08598b7000-7f08598bc000 rw-p 00000000 00:00 0
7f08598bc000-7f08598be000 r-xp 00000000 08:02 23896079                   /lib64/libdl-2.11.1.so
7f08598be000-7f0859abe000 ---p 00002000 08:02 23896079                   /lib64/libdl-2.11.1.so
7f0859abe000-7f0859abf000 r--p 00002000 08:02 23896079                   /lib64/libdl-2.11.1.so
7f0859abf000-7f0859ac0000 rw-p 00003000 08:02 23896079                   /lib64/libdl-2.11.1.so
7f0859ac0000-7f0859acc000 r-xp 00000000 08:02 23896077                   /lib64/libcrypt-2.11.1.so
7f0859acc000-7f0859ccb000 ---p 0000c000 08:02 23896077                   /lib64/libcrypt-2.11.1.so
7f0859ccb000-7f0859ccc000 r--p 0000b000 08:02 23896077                   /lib64/libcrypt-2.11.1.so
7f0859ccc000-7f0859ccd000 rw-p 0000c000 08:02 23896077                   /lib64/libcrypt-2.11.1.so
7f0859ccd000-7f0859cfb000 rw-p 00000000 00:00 0
7f0859cfb000-7f0859d03000 r-xp 00000000 08:02 23896103                   /lib64/librt-2.11.1.so
7f0859d03000-7f0859f02000 ---p 00008000 08:02 23896103                   /lib64/librt-2.11.1.so
7f0859f02000-7f0859f03000 r--p 00007000 08:02 23896103                   /lib64/librt-2.11.1.so
7f0859f03000-7f0859f04000 rw-p 00008000 08:02 23896103                   /lib64/librt-2.11.1.so
7f0859f04000-7f0859f59000 r-xp 00000000 08:02 23896081                   /lib64/libm-2.11.1.so
7f0859f59000-7f085a158000 ---p 00055000 08:02 23896081                   /lib64/libm-2.11.1.so
7f085a158000-7f085a159000 r--p 00054000 08:02 23896081                   /lib64/libm-2.11.1.so
7f085a159000-7f085a15a000 rw-p 00055000 08:02 23896081                   /lib64/libm-2.11.1.so
7f085a15a000-7f085a15b000 r-xp 00000000 08:02 23896155                   /lib64/libaio.so.1.0.1
7f085a15b000-7f085a35a000 ---p 00001000 08:02 23896155                   /lib64/libaio.so.1.0.1
7f085a35a000-7f085a35b000 r--p 00000000 08:02 23896155                   /lib64/libaio.so.1.0.1
7f085a35b000-7f085a35c000 rw-p 00001000 08:02 23896155                   /lib64/libaio.so.1.0.1
7f085a35c000-7f085a373000 r-xp 00000000 08:02 23896099                   /lib64/libpthread-2.11.1.so
7f085a373000-7f085a573000 ---p 00017000 08:02 23896099                   /lib64/libpthread-2.11.1.so
7f085a573000-7f085a574000 r--p 00017000 08:02 23896099                   /lib64/libpthread-2.11.1.so
7f085a574000-7f085a575000 rw-p 00018000 08:02 23896099                   /lib64/libpthread-2.11.1.so
7f085a575000-7f085a579000 rw-p 00000000 00:00 0
7f085a579000-7f085a598000 r-xp 00000000 08:02 23896066                   /lib64/ld-2.11.1.soshow processlist;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
ERROR:
Can't connect to the server
[7 Feb 2012 20:57] Flavian C
------------------------------------------------------------------------------

when i saw the .err file this was the report

mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=1
max_threads=200
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6998246 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7796bf0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f077d05d858 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7c2729]
/usr/sbin/mysqld(handle_segfault+0x3c2)[0x518652]
/lib64/libpthread.so.0(+0xf5d0)[0x7f085a36b5d0]
/lib64/libc.so.6(gsignal+0x35)[0x7f0859590945]
/lib64/libc.so.6(abort+0x181)[0x7f0859591f21]
/lib64/libc.so.6(+0x6f8ef)[0x7f08595cd8ef]
/lib64/libc.so.6(+0x75018)[0x7f08595d3018]
/lib64/libc.so.6(cfree+0x6c)[0x7f08595d7f6c]
/usr/sbin/mysqld(_ZN15Query_log_eventD0Ev+0x28)[0x73e558]
/usr/sbin/mysqld[0x53523c]
/usr/sbin/mysqld(handle_slave_sql+0xada)[0x53777a]
/lib64/libpthread.so.0(+0x75f0)[0x7f085a3635f0]
/lib64/libc.so.6(clone+0x6d)[0x7f085963284d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query ((nil)): is an invalid pointer
Connection ID (thread ID): 10
Status: NOT_KILLED
[8 Feb 2012 4:48] Valeriy Kravchuk
Please, send the output of:

uname -a
free

from your system.
[8 Feb 2012 6:48] Flavian C
product1-slave:~ # uname -a
Linux product1-slave 2.6.32.12-0.7-default #1 SMP 2010-05-20 11:14:20 +0200 x86_64 x86_64 x86_64 GNU/Linux
product1-slave:~ # free
             total       used       free     shared    buffers     cached
Mem:       5659556    1421668    4237888          0      36876     539780
-/+ buffers/cache:     845012    4814544
Swap:      9181108          0    9181108
[8 Feb 2012 18:07] Flavian C
any update????
[10 Feb 2012 15:22] Flavian C
any progress on the bug...?????
[28 Feb 2012 13:52] Valeriy Kravchuk
No progress, yet, as it is unclear how to repeat this and I suspect hardware failure more than anything else.

Please, upload the entire error log from problematic slave, compressed. Do you still have master's binary log that slave was working with at the moment?
[14 Mar 2012 12:23] Flavian C
Nope i dont have the binlog file... its deleted every 3days.
[20 Mar 2012 16:39] Sveta Smirnova
Thank you for the feedback.

Looks like we need to wait when issue happens next time.

If it does, please:

- check for hardware failure (RAM and the disk where MySQL files are located)
If hardware is OK:
- keep, then send it to us, error log file from slave
- keep, then send it to us, master's binary log file which was replicated at the time when issue happened
[21 Apr 2012 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".