MySQL Bugs: #62942: double free or corruption when replication

Bug #62942	double free or corruption when replication
Submitted:	26 Oct 2011 9:27	Modified:	30 Apr 2012 5:48
Reporter:	Shintaro Nozaki	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server: Replication	Severity:	S2 (Serious)
Version:	5.5.17	OS:	Linux
Assigned to:		CPU Architecture:	Any
Tags:	replication

Description:
environments as follows:

OS : CentOS 6.0 on VMWare Fuction(Mac OS X 10.6.8)
MySQL : MySQL-client-5.5.17-1.linux2.6.i386, MySQL-devel-5.5.17-1.linux2.6.i386, MySQL-server-5.5.17-1.linux2.6.i386, MySQL-shared-5.5.17-1.linux2.6.i386 
CPU : Intel(R) Core(TM) i7 CPU 870  @ 2.93GHz
Memory allocation : 2GB(Mac is 8GB)
(Slave is the same as Master)

my.cnf of Master as follows:

[mysqld]
skip-character-set-client-handshake
character-set-serve                 = utf8
character-set-filesystem            = utf8
datadir                             = /var/lib/mysql/
log-error                           = /home/mysql/mysqld.err
log-bin                             = /home/mysql/log-bin
master-info-file                    = /home/mysql/master.info
server-id                           = 1
binlog-do-db=hoge
binlog_format                       = STATEMENT

[mysql]
default-character-set               = utf8

[mysqldump]
default-character-set               = utf8
hex-blob

my.cnf of Slave as follows:

[mysqld]
skip-character-set-client-handshake
character-set-serve                 = utf8
character-set-filesystem            = utf8
datadir                             = /var/lib/mysql/
log-error                           = /home/mysql/mysqld.err
relay-log                           = /home/mysql/relay-bin
relay-log-info-file                 = /home/mysql/relay-log.info
server-id                           = 2
replicate-rewrite-db=hoge->foobarhogefoofoofoofoofoofoo
replicate-do-db=foobarhogefoofoofoofoofoofoo
replicate-wild-do-table=foobarhogefoofoofoofoofoofoo.baz
query_cache_size                    = 1M

[mysql]
default-character-set               = utf8

[mysqldump]
default-character-set               = utf8
hex-blob

error on Slave as follows:

 *** glibc detected *** /usr/sbin/mysqld: double free or corruption (!prev): 0xa4b09a50 ***
======= Backtrace: =========
/lib/libc.so.6(+0x6f1a1)[0x17f1a1]
/usr/sbin/mysqld(my_free+0x1d)[0x83bfaed]
/usr/sbin/mysqld(_ZN15Query_log_eventD0Ev+0x30)[0x8343206]
/usr/sbin/mysqld[0x813c3d0]
/usr/sbin/mysqld(handle_slave_sql+0xc4e)[0x813ea0f]
/lib/libpthread.so.0(+0x69e9)[0x4249e9]
/lib/libc.so.6(clone+0x5e)[0x1eddfe]
======= Memory map: ========
00110000-00295000 r-xp 00000000 fd:00 523149     /lib/libc-2.12.so
00295000-00297000 r--p 00185000 fd:00 523149     /lib/libc-2.12.so
00297000-00298000 rw-p 00187000 fd:00 523149     /lib/libc-2.12.so
00298000-0029b000 rw-p 00000000 00:00 0 
0029b000-002e0000 r-xp 00000000 fd:00 523129     /lib/libfreebl3.so
002e0000-002e1000 rw-p 00045000 fd:00 523129     /lib/libfreebl3.so
002e1000-002e5000 rw-p 00000000 00:00 0 
0041e000-00435000 r-xp 00000000 fd:00 523173     /lib/libpthread-2.12.so
00435000-00436000 r--p 00016000 fd:00 523173     /lib/libpthread-2.12.so
00436000-00437000 rw-p 00017000 fd:00 523173     /lib/libpthread-2.12.so
00437000-00439000 rw-p 00000000 00:00 0 
007fc000-00808000 r-xp 00000000 fd:00 523165     /lib/libnss_files-2.12.so
00808000-00809000 r--p 0000b000 fd:00 523165     /lib/libnss_files-2.12.so
00809000-0080a000 rw-p 0000c000 fd:00 523165     /lib/libnss_files-2.12.so
008c9000-008e6000 r-xp 00000000 fd:00 523126     /lib/libgcc_s-4.4.4-20100726.so.1
008e6000-008e7000 rw-p 0001d000 fd:00 523126     /lib/libgcc_s-4.4.4-20100726.so.1
00a0d000-00a0e000 r-xp 00000000 00:00 0          [vdso]
00b7e000-00b7f000 r-xp 00000000 fd:00 523528     /lib/libaio.so.1.0.1
00b7f000-00b80000 rw-p 00000000 fd:00 523528     /lib/libaio.so.1.0.1
00bbd000-00bdb000 r-xp 00000000 fd:00 523142     /lib/ld-2.12.so
00bdb000-00bdc000 r--p 0001d000 fd:00 523142     /lib/ld-2.12.so
00bdc000-00bdd000 rw-p 0001e000 fd:00 523142     /lib/ld-2.12.so
00c6d000-00c74000 r-xp 00000000 fd:00 523177     /lib/librt-2.12.so
00c74000-00c75000 r--p 00006000 fd:00 523177     /lib/librt-2.12.so
00c75000-00c76000 rw-p 00007000 fd:00 523177     /lib/librt-2.12.so
00c76000-00c9e000 r-xp 00000000 fd:00 523157     /lib/libm-2.12.so
00c9e000-00c9f000 r--p 00027000 fd:00 523157     /lib/libm-2.12.so
00c9f000-00ca0000 rw-p 00028000 fd:00 523157     /lib/libm-2.12.so
00d1b000-00d22000 r-xp 00000000 fd:00 523153     /lib/libcrypt-2.12.so
00d22000-00d23000 r--p 00007000 fd:00 523153     /lib/libcrypt-2.12.so
00d23000-00d24000 rw-p 00008000 fd:00 523153     /lib/libcrypt-2.12.so
00d24000-00d4b000 rw-p 00000000 00:00 0 
00d7e000-00d81000 r-xp 00000000 fd:00 523155     /lib/libdl-2.12.so
00d81000-00d82000 r--p 00002000 fd:00 523155     /lib/libdl-2.12.so
00d82000-00d83000 rw-p 00003000 fd:00 523155     /lib/libdl-2.12.so
08048000-08889000 r-xp 00000000 fd:00 795698     /usr/sbin/mysqld
08889000-08951000 rw-p 00840000 fd:00 795698     /usr/sbin/mysqld
08951000-08971000 rw-p 00000000 00:00 0 
09093000-09808000 rw-p 00000000 00:00 0          [heap]
a4a00000-a4a21000 rw-p 00000000 00:00 0 
a4a21000-a4b00000 ---p 00000000 00:00 0 
a4b00000-a4b21000 rw-p 00000000 00:00 0 
a4b21000-a4c00000 ---p 00000000 00:00 0 
a4c00000-a4c2e000 rw-p 00000000 00:00 0 
a4c2e000-a4d00000 ---p 00000000 00:00 0 
a4d45000-a4d46000 ---p 00000000 00:00 0 
a4d46000-a5746000 rw-p 00000000 00:00 0 
a5746000-a5747000 ---p 00000000 00:00 0 
a5747000-a6147000 rw-p 00000000 00:00 0 
a6147000-a6148000 ---p 00000000 00:00 0 
a6148000-a6b48000 rw-p 00000000 00:00 0 
a6b48000-a6b49000 ---p 00000000 00:00 0 
a6b49000-a77d7000 rw-p 00000000 00:00 0 
a7900000-a7921000 rw-p 00000000 00:00 0 
a7921000-a7a00000 ---p 00000000 00:00 0 
a7a85000-a7a86000 ---p 00000000 00:00 0 
a7a86000-a7ab6000 rw-p 00000000 00:00 0 
a7ab6000-a7ab7000 ---p 00000000 00:00 0 
a7ab7000-a7ae7000 rw-p 00000000 00:00 0 
a7ae7000-a7ae8000 ---p 00000000 00:00 0 
a7ae8000-a84e8000 rw-p 00000000 00:00 0 
a84e8000-a84e9000 ---p 00000000 00:00 0 
a84e9000-a8ee9000 rw-p 00000000 00:00 0 
a8ee9000-a8eea000 ---p 00000000 00:00 0 
a8eea000-a98ea000 rw-p 00000000 00:00 0 
a98ea000-a98eb000 ---p 00000000 00:00 0 
a98eb000-aa2eb000 rw-p 00000000 00:00 0 
aa2eb000-aa2ec000 ---p 00000000 00:00 0 
aa2ec000-aacec000 rw-p 00000000 00:00 0 
aacec000-aaced000 ---p 00000000 00:00 0 
aaced000-ab6ed000 rw-p 00000000 00:00 0 
ab6ed000-ab6ee000 ---p 00000000 00:00 0 
ab6ee000-ac0ee000 rw-p 00000000 00:00 0 
ac0ee000-ac0ef000 ---p 00000000 00:00 0 
ac0ef000-acaef000 rw-p 00000000 00:00 0 
acaef000-acaf0000 ---p 00000000 00:00 0 
acaf0000-b6c00000 rw-p 00000000 00:00 0 
b6c00000-b6c21000 rw-p 00000000 00:00 0 
b6c21000-b6d00000 ---p 00000000 00:00 0 
b6d0f000-b6d10000 ---p 00000000 00:00 0 
b6d10000-b6d40000 rw-p 00000000 00:00 0 
b6d40000-b6d41000 ---p 00000000 00:00 0 
b6d41000-b6d71000 rw-p 00000000 00:00 0 
b6d71000-b6d72000 ---p 00000000 00:00 0 
b6d72000-b7775000 rw-p 00000000 00:00 0 
b7782000-b7784000 rw-p 00000000 00:00 0 
bfd4a000-bfd5f000 rw-p 00000000 00:00 0          [stack]

How to repeat:
1. Slave : stop replication

stop slave;
reset slave;

2. Master : create database and table

create database hoge;
use hoge;
CREATE TABLE `baz` (
 `id` int(10) NOT NULL AUTO_INCREMENT,
`name` varchar(200),
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
reset master;

3. Slave : create database and table

create database foobarhogefoofoofoofoofoofoo;
use foobarhogefoofoofoofoofoofoo;
CREATE TABLE `baz` (
 `id` int(10) NOT NULL AUTO_INCREMENT,
`name` varchar(200),
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
reset slave;

4. Slave : start slave

start slave;

5. Master : run query

use hoge;
/*!40000 ALTER TABLE `baz` DISABLE KEYS */;

6. Slave : mysql restart

$ sudo /etc/init.d/mysql restart

Possibly related to https://bugs.launchpad.net/percona-server/+bug/705688
Temp fix (until bug is fixed) = turn off query_cache_strip_comments

Thank you for the report.

I can not repeat described behavior with current version 5.5.19. Please try with this version and inform us if problem still exists.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

I'm having this same problem on MySQL 5.5.20, query_Cache_strip_comments is off.  :(

Sheeri,

> I'm having this same problem on MySQL 5.5.20, query_Cache_strip_comments is off.  :(

Which server do you use? Can you repeat same behavior with MySQL's distribution?

See also bug #64624

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Sveta, please run the slave in valgrind.  Then we have a repeatable problem with query cache, same as seen in bug #62942 .  The steps to repeat it here are accurate enough, so I'm setting this verified.

5.5.23:

Thread 19:
Conditional jump or move depends on uninitialised value(s)
at: Query_cache::send_result_to_client (sql_cache.cc:1520)
by: mysql_parse (sql_parse.cc:5572)
by: Query_log_event::do_apply_event (log_event.cc:3396)
by: apply_event_and_update_pos(log_event.h:1132)
by: exec_relay_log_event (slave.cc:2553)
by: handle_slave_sql (slave.cc:3370)
by: start_thread (pthread_create.c:301)
by: clone (clone.S:115)
Uninitialised value was created by a heap allocation
at: malloc (vg_replace_malloc.c:263)
by: my_malloc (my_malloc.c:38)
by: Query_log_event::Query_log_event (log_event.cc:2914)
by: Log_event::read_log_event (log_event.cc:1257)
by: Log_event::read_log_event (log_event.cc:1170)
by: exec_relay_log_event (slave.cc:4607)
by: handle_slave_sql (slave.cc:3370)
by: start_thread (pthread_create.c:301)
by: clone (clone.S:115)

Thank you to Shintaro for testcase.  Let's call it duplicate of bug #64624.  We already have an internal bug #14005409 for that, but not for this.

I ran into this error on my server. In case it helps anyone, it seemed to have been caused (in my case) due to corrupt ARCHIVE engine tables. 

Once I removed the corrupt tables, the error disappeared.