MySQL Bugs: #54053: Drop table mysql.ibbackup_binlog_marker crashes the server for innodb plugin

Bug #54053	Drop table mysql.ibbackup_binlog_marker crashes the server for innodb plugin
Submitted:	28 May 2010 3:15	Modified:	5 Oct 2010 20:06
Reporter:	Hema Sridharan	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Enterprise Backup	Severity:	S2 (Serious)
Version:	mysql-5.1-meb, MEB 3.1.0	OS:	Linux
Assigned to:	Thava Alagu	CPU Architecture:	Any

Description:
* When innobackup executes with --include option, mysql.ibbackup_binlog_marker table gets created. 
* When we try to drop this table for innodb plugin, MySQL server crashes

Note that the crash does not happen when we don't backup using the --include option.

How to repeat:
* Please see the attached test to reproduce the crash.

Here is the stack trace for the crash

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60559 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x58ea4e8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4986d100 thread_stack 0x40000
(my_print_stacktrace+0x32)[0xb05f72]
(handle_segfault+0x28a)[0x698974]
/lib64/libpthread.so.0[0x36a3c0de60]
/lib64/libc.so.6(gsignal+0x35)[0x36a3030045]
/lib64/libc.so.6(abort+0x110)[0x36a3031ae0]
[0x2aaaaac43f5a]
[0x2aaaaac3befa]
[0x2aaaaac323ac]
[0x2aaaaacadded]
[0x2aaaaac5d0e8]
(_ZN7handler15ha_delete_tableEPKc+0x33)[0x7e84fb]
(_Z15ha_delete_tableP3THDP10handlertonPKcS4_S4_b+0x171)[0x7ee481]
(_Z20mysql_rm_table_part2P3THDP10TABLE_LISTbbbb+0x89e)[0x81955e]
(_Z14mysql_rm_tableP3THDP10TABLE_LISTcc+0x116)[0x819b22]
(_Z21mysql_execute_commandP3THD+0x3e38)[0x6ad5dc]
(_Z11mysql_parseP3THDPKcjPS2_+0x2a5)[0x6b2b53]
(_Z16dispatch_command19enum_server_commandP3THDPcj+0xbe0)[0x6b4314]
(_Z10do_commandP3THD+0x252)[0x6b5762]
(handle_one_connection+0x13d)[0x6a1ba9]
/lib64/libpthread.so.0[0x36a3c062e7]
/lib64/libc.so.6(clone+0x6d)[0x36a30ce3bd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x59466d8 = DROP TABLE IF EXISTS mysql.ibbackup_binlog_marker
thd->thread_id=1
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

Here is the attached test case to reproduce the crash

Attachment: innobackup_bug.test (application/test, text), 3.75 KiB.

Thank you for the report.

Verified as described. Option --innodb-file-per-table=1 should be used.

The attached test case removes datadir/dbb directory when the server is up and running. This is not supported. It will be interesting to see if the bug reproduces if the rm -rf datadir/dbb is replaced with drop database dbb;

It still crashes with InnoDB Plugin

Need to fix the reproducible test case that performs unsupported operation.

I was able to reproduce the crash. The root cause of the problem seems to be with the innodb plugin which crashes after detecting a missing tablespace (.ibd file).

However the same problem does not reproduce with the latest innodb plugin available with mysql-5.1 tree.

The problem is reproducible with mysql-5.1-meb tree as of today which contains changes from mysql-5.1 upto 2010-04-12; One of the later changes has changed the way of table space id assignment and management. May be the fix for Bug #53578 [ assert on invalid page access, in fil_io() ] may have fixed this problem too.

The crash produces the following stack trace:

#0  0xb8034430 in __kernel_vsyscall ()
#1  0xb7ff1158 in pthread_kill () from /lib/tls/i686/cmov/libpthread.so.0
#2  0x0860c87d in my_write_core (sig=6) at stacktrace.c:329
#3  0x08282995 in handle_segfault (sig=6) at mysqld.cc:2571
#4  <signal handler called>
#5  0xb8034430 in __kernel_vsyscall ()
#6  0xb7e278a0 in raise () from /lib/tls/i686/cmov/libc.so.6
#7  0xb7e29268 in abort () from /lib/tls/i686/cmov/libc.so.6
#8  0xb74468ec in fil_open_single_table_tablespace (check_space_id=1, id=1, flags=1, 
    name=0xa96a3c0c "mydb/t1") at fil/fil0fil.c:2981
#9  0xb743e6f7 in dict_load_table (name=0xa96a3c0c "mydb/t1") at dict/dict0load.c:974
#10 0xb7434d7e in dict_table_get_low (table_name=0xa96a3c0c "mydb/t1")
    at ./include/dict0dict.ic:768
#11 0xb74ad60d in row_drop_table_for_mysql (name=0xa96a3c0c "mydb/t1", trx=0x9d5e140, drop_db=0)
    at row/row0mysql.c:3050
#12 0xb745f788 in ha_innodb::delete_table (this=0x9d875b0, name=0xa96a4e9b "./mydb/t1")
    at handler/ha_innodb.cc:6911
#13 0x083c9bda in handler::ha_delete_table (this=0x9d875b0, name=0xa96a4e9b "./mydb/t1")
    at handler.cc:3369
#14 0x083d0476 in ha_delete_table (thd=0x9d5a3f0, table_type=0x95a4bc8, 
    path=0xa96a4e9b "./mydb/t1", db=0x9d875a8 "mydb", alias=0x9d873a8 "t1", 
    generate_warning=true) at handler.cc:1992
#15 0x083fb824 in mysql_rm_table_part2 (thd=0x9d5a3f0, tables=0x9d873d0, if_exists=false, 
    drop_temporary=false, drop_view=false, dont_log_query=false) at sql_table.cc:2040
#16 0x083fbf5a in mysql_rm_table (thd=0x9d5a3f0, tables=0x9d873d0, if_exists=0 '\0', 
    drop_temporary=0 '\0') at sql_table.cc:1819
#17 0x082993ef in mysql_execute_command (thd=0x9d5a3f0) at sql_parse.cc:3385
#18 0x0829f33b in mysql_parse (thd=0x9d5a3f0, inBuf=0x9d87340 "DROP TABLE t1", length=13, 
    found_semicolon=0xa96a6090) at sql_parse.cc:5971
#19 0x082a015f in dispatch_command (command=COM_QUERY, thd=0x9d5a3f0, 
    packet=0x9d7f2e1 "DROP TABLE t1", packet_length=13) at sql_parse.cc:1233
#20 0x082a16c4 in do_command (thd=0x9d5a3f0) at sql_parse.cc:874
#21 0x0828c77b in handle_one_connection (arg=0x9d5a3f0) at sql_connect.cc:1127
#22 0xb7fec50f in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#23 0xb7edda0e in clone () from /lib/tls/i686/cmov/libc.so.6

The crash happens in  fil_open_single_table_tablespace() function line 2981 assert failure:
   ut_a(flags != DICT_TF_COMPACT);

Following MTR test script rm_ibd.test can be used to reproduce the problem.

Note that missing tablespace condition should not cause crash; For documentation on how to deal with such conditions, see:  
http://dev.mysql.com/doc/refman/5.1/en/innodb-troubleshooting-datadict.html

--source include/not_embedded.inc
--source include/have_innodb.inc
--source suite/innodb/include/have_innodb_plugin.inc
SET GLOBAL innodb_file_per_table=on;

show plugins;
show variables like '%innodb%' ;

LET $MYSQLD_DATADIR = `SELECT @@datadir`;

CREATE DATABASE mydb;
CREATE TABLE mydb.t1(id INT)ENGINE=INNODB;
INSERT INTO mydb.t1 VALUES (1),(2),(3),(4),(5);

# Now shutdown the server.
--source suite/meb/include/stop_mysqld.inc

# Now delete  mydb/t1.ibd file
--remove_file $MYSQLD_DATADIR/mydb/t1.ibd

# Restart server.
--source suite/meb/include/restart_mysqld.inc

# After restore verify that databases and tables are restored
USE mydb;
# SHOW TABLES;
DROP TABLE t1;

exit;

rm_ibd.opt file contains:
--ignore-builtin-innodb

The problem may have been fixed by now, but it needs review by some one from innodb plugin development team.

I think this was fixed by http://bugs.mysql.com/bug.php?id=54009

Verified that the problem no longer reproduces with the latest mysql 5.1 version.

Verified this bug in the latest mysql-5.1 tree and seems the issue is fixed now. So closing the bug.