MySQL Bugs: #39056: Server crash for concurrent backup operations with 1 of them backup to pipe

Bug #39056	Server crash for concurrent backup operations with 1 of them backup to pipe
Submitted:	26 Aug 2008 21:50	Modified:	11 Nov 2008 15:56
Reporter:	Hema Sridharan	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: Backup	Severity:	S2 (Serious)
Version:	mysql-6.0-backup, mysql-6.0	OS:	Linux
Assigned to:	Chuck Bell	CPU Architecture:	Any

Description:
1) I create databases(db1 and db2), tables t1.db1 and t2.db2 and insert
some contents in table.
2) Perform backup database db1 from connection 1 using pipe. Also start
backup of db2 from connection 2 simultaneously.
3) I then read the backup data from pipe and send it to the local file 
 (cat /export/home/tmp/backup_dmp/db1_pipe > /export/home/tmp/backup_dmp/db1_file ). Immediately noticed the Mysql Server crash.
4) Server crash occurs only when we try to execute another backup
operation in parallel from connection2.

Note: Generally when we have 2 backup operations going on simultaneously from
say connection 1 and connection 2, the first backup operation completes
successfully and 2nd operation shows error message as "ERROR 1627
(HY000): Can't execute this command because another BACKUP/RESTORE
operation is in progress". But in case of backup to pipe from one con1
followed by another backup in different connection, leads to server crash.

How to repeat:
create a pipe 
mkfifo /export/home/tmp/backup_dmp/db1_pipe
chmod 666 /export/home/tmp/backup_dmp/db1_pipe

connection con1;
CREATE DATABASE db1;
CREATE DATABASE db2;
USE db1;
CREATE TABLE t1(id int);
INSERT INTO t1 VALUES(1),(2),(3),(4),(5),(6),(7),(8),(9);
INSERT INTO t1 SELECT * FROM t1;
INSERT INTO t1 SELECT * FROM t1;
INSERT INTO t1 SELECT * FROM t1;
INSERT INTO t1 SELECT * FROM t1;
USE db2;
CREATE TABLE t2(a char(5));
INSERT INTO t2 VALUES('a'),('b'),('c'),('d'),('e'),('f'),('g'),('h');
INSERT INTO t2 SELECT * FROM t2;
INSERT INTO t2 SELECT * FROM t2;
INSERT INTO t2 SELECT * FROM t2;

BACKUP DATABASE db1 to '/export/home/tmp/backup_dmp/db1_pipe';

connection con2;

BACKUP DATABASE db2 to '/export/home/tmp/backup_dmp/db2.bak';
ERROR 1627 (HY000): Can't execute this command because another BACKUP/RESTORE operation is in progress

on another console,
Reading the backup data from pipe, immediately results in server crash.
cat /export/home/tmp/backup_dmp/db1_pipe > /export/home/tmp/backup_dmp/db1_file

connection con1;

mysql> BACKUP DATABASE db1 to '/export/home/tmp/backup_dmp/db1_pipe';
ERROR 2013 (HY000): Lost connection to MySQL server during query

STACK TRACE
===========
080827  0:34:36 [Note] Backup: Backup completed
mysqld: kernel.cc:1197: bstream_byte* bstream_alloc(long unsigned int): Assertion `Backup_restore_ctx::mem_alloc' failed.
/home/hs221732/team6.0p/libexec/mysqld(my_print_stacktrace+0x32)[0xd41e46]
/home/hs221732/team6.0p/libexec/mysqld(handle_segfault+0x2a6)[0x6ff692]
/lib64/libpthread.so.0[0x36a3c0de60]
/lib64/libc.so.6(gsignal+0x35)[0x36a3030045]
/lib64/libc.so.6(abort+0x110)[0x36a3031ae0]
/lib64/libc.so.6(__assert_fail+0xf6)[0x36a3029756]
/home/hs221732/team6.0p/libexec/mysqld(bstream_alloc+0x3a)[0xdd9d16]
/home/hs221732/team6.0p/libexec/mysqld(bstream_open_wr+0x51)[0xdf3282]
/home/hs221732/team6.0p/libexec/mysqld(_ZN6backup13Output_stream4initEv+0x6e)[0xdec4fe]
/home/hs221732/team6.0p/libexec/mysqld(_ZN6backup13Output_stream4openEv+0x26b)[0xdec7ef]
/home/hs221732/team6.0p/libexec/mysqld(_ZN18Backup_restore_ctx18prepare_for_backupE19st_mysql_lex_stringPKcb+0x130)[0xddb3fa]
/home/hs221732/team6.0p/libexec/mysqld(_Z22execute_backup_commandP3THDP6st_lex+0x169)[0xddb89d]
/home/hs221732/team6.0p/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x96a)[0x71118e]
/home/hs221732/team6.0p/libexec/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x275)[0x71966b]
/home/hs221732/team6.0p/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x9c5)[0x71a1c1]
/home/hs221732/team6.0p/libexec/mysqld(_Z10do_commandP3THD+0x220)[0x71b670]
/home/hs221732/team6.0p/libexec/mysqld(handle_one_connection+0x11a)[0x708314]
/lib64/libpthread.so.0[0x36a3c062e7]
/lib64/libc.so.6(clone+0x6d)[0x36a30ce3bd]
080827  0:34:47 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=2
max_threads=151
thread_count=2
connection_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60646 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x2aaaac8c1918
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4be940f0 thread_stack 0x40000
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa5bbc80 = BACKUP DATABASE db1 to '/export/home/tmp/backup_dmp/db1_pipe'
thd->thread_id=1
thd->killed=NOT_KILLED

Thank you for the bug report. Verified as described.

Using the latest code, the scenario described does not crash. Instead, the backup can be completed without errors and works as expected.

Sample test and result file uploaded.

Test file showing problem does not exist.

Attachment: bug_39056.test (application/octet-stream, text), 1.83 KiB.

Result file.

Attachment: bug_39056.result (application/octet-stream, text), 955 bytes.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65398

2761 Hema Sridharan	2009-02-05
      BUG#39056. This test verifies that there is 
      no server crash because of concurrent backup / restore operations when one of them is backup to pipe.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65669

2761 Hema Sridharan	2009-02-09
      BUG#39056 Test(backup_pipe.test) created for this defect which will help for regression.
      Modified the test as per requests from review comments.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/66089

2771 Hema Sridharan	2009-02-12
      BUG#39056(backup_pipe.test) Test created for this defect which will help for regression. This patch will be now compatible with new mtr.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/66126

2770 Hema Sridharan	2009-02-13
      Bug#39056(backup_pipe.test) This test is created for regression purpose.

Pushed into 6.0.11-alpha (revid:alik@sun.com-20090326121822-pt84kzxxayzho4mn) (version source revid:rafal.somla@sun.com-20090302164601-znhm4tadplfi2iqu) (merge vers: 6.0.11-alpha) (pib:6)