Bug #70007 Missing tables in InnoDB dictionary cause assertion and restart of MySQL
Submitted: 13 Aug 2013 3:13 Modified: 18 Sep 2013 19:15
Reporter: Justin Swanhart Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: DDL Severity:S1 (Critical)
Version:5.6.13 OS:Any
Assigned to: CPU Architecture:Any

[13 Aug 2013 3:13] Justin Swanhart
Description:
Version: '5.6.13'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2013-08-12 16:42:38 11316 [Warning] InnoDB: Orphan table encountered during DROP DATABASE. This is possible if 'test/aggregation_tmp_21730619.frm' was lost.
2013-08-12 16:42:38 7fbfe1399700  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2013-08-12 16:42:38 7fbfe1399700  InnoDB: Fatal error: cannot open ./test/aggregation_tmp_21730619.ibd
.InnoDB: Have you deleted .ibd files under a running mysqld server?
2013-08-12 16:42:38 7fbfe1399700  InnoDB: Assertion failure in thread 140462094128896 in file fil0fil.cc line 740
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
23:42:38 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=46
max_threads=151
thread_count=13
connection_count=13
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 191105 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x34cf9c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fbfe1398e18 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8c90d5]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x65cee4]
/lib64/libpthread.so.0(+0xf500)[0x7fbfe3e4f500]
/lib64/libc.so.6(gsignal+0x35)[0x7fbfe2afa8a5]
/lib64/libc.so.6(abort+0x175)[0x7fbfe2afc085]
/usr/sbin/mysqld[0xa2ef94]
/usr/sbin/mysqld[0xa2f095]
/usr/sbin/mysqld[0xa2f2cc]
/usr/sbin/mysqld[0xa05bd6]
/usr/sbin/mysqld[0x98c670]
/usr/sbin/mysqld[0x98dab1]
/usr/sbin/mysqld[0x94df58]
/usr/sbin/mysqld[0x94e465]
/usr/sbin/mysqld[0x974a68]
/usr/sbin/mysqld[0x9753f8]
/usr/sbin/mysqld[0x8ed379]
/usr/sbin/mysqld[0x59f9cf]
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x216)[0x6e2e26]
/usr/sbin/mysqld(_Z11mysql_rm_dbP3THDPcbb+0x912)[0x6acb12]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x35ad)[0x6d971d]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x32f)[0x6daecf]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xe5c)[0x6dbe1c]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0xcf)[0x6a97af]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6a98d7]
/usr/sbin/mysqld(pfs_spawn_thread+0x12a)[0xaec39a]

How to repeat:
These errors used to just print an error in the log but now they cause a server to restart.  Sometimes orphaned tables happen, especially in systems that were 5.5 or earlier where deleting .ibd and .frm files didn't have such a bad impact.

Suggested fix:
Print warning messages instead of asserting.
[13 Aug 2013 13:08] Umesh Shastry
Hello Justin,

Thank you for the bug report. 
Verified as described.

Thanks,
Umesh
[13 Aug 2013 13:09] Umesh Shastry
// 5.6.13

How to repeat:

use test;
drop table keyvalue1;
CREATE TABLE `test`.`keyvalue1` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(40),
  PRIMARY KEY (`id`)
) ENGINE=Innodb;

set @id:=0;

 insert into `keyvalue1` values (@id:=@id+1,md5(rand()*1000000)),(@id:=@id+1,md5(rand()*1000000)),(@id:=@id+1,md5(rand()*1000000)),(@id:=@id+1,md5(rand()*1000000));

 insert into `keyvalue1`(`id`,`name`)
   select @id:=@id+1,md5(rand()*1000000) from
    `keyvalue1` k1, `keyvalue1` k2, `keyvalue1` k3, `keyvalue1` k4,`keyvalue1` k5,`keyvalue1` k6, `keyvalue1` k7, `keyvalue1` k8, `keyvalue1` k9,
`keyvalue1` k0,`keyvalue1` ka, `keyvalue1` kb, `keyvalue1` kc, `keyvalue1` kd limit
50000; 

// Remove keyvalue1.frm while MySQL server is running

- rm -rf /tmp/<datadir>/test/*.frm

- Bring down MySQL server, ensure it is shutdown, bring up MySQL server again( I used mysqld-debug in my tests )

// Remove keyvalue1.ibd

- rm -rf /tmp/<datadir>/test/*.ibd

- try to drop database "test" 

mysql> drop database test;
ERROR 2013 (HY000): Lost connection to MySQL server during query

//  Check error log
..
..
2013-08-14 16:38:51 24299 [Warning] InnoDB: Orphan table encountered during DROP DATABASE. This is possible if 'test/keyvalue1.frm' was lost.
2013-08-14 16:38:51 7fc2e80d2700  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2013-08-14 16:38:51 7fc2e80d2700  InnoDB: Fatal error: cannot open ./test/keyvalue1.ibd
.InnoDB: Have you deleted .ibd files under a running mysqld server?
2013-08-14 16:38:51 7fc2e80d2700  InnoDB: Assertion failure in thread 140475093559040 in file fil0fil.cc line 740
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
11:08:51 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1
max_threads=2000
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4387395 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x317bdb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc2e80d1de8 thread_stack 0x40000
bin/mysqld-debug(my_print_stacktrace+0x32)[0xa8fb70]
bin/mysqld-debug(handle_fatal_signal+0x47a)[0x70295a]
/lib64/libpthread.so.0[0x3b2560f500]
/lib64/libc.so.6(gsignal+0x35)[0x3b24e328a5]
/lib64/libc.so.6(abort+0x175)[0x3b24e34085]
bin/mysqld-debug[0xcb9173]
bin/mysqld-debug[0xcb9844]
bin/mysqld-debug[0xcb9a0b]
bin/mysqld-debug[0xcb9aa1]
bin/mysqld-debug[0xcb9b03]
bin/mysqld-debug[0xc855dc]
bin/mysqld-debug[0xbccbd8]
bin/mysqld-debug[0xbcd08f]
bin/mysqld-debug[0xbcd4f0]
bin/mysqld-debug[0xb539b1]
bin/mysqld-debug[0xb53c80]
bin/mysqld-debug[0xb53e28]
bin/mysqld-debug[0xb548c2]
bin/mysqld-debug[0xb95dad]
bin/mysqld-debug[0xb969c0]
bin/mysqld-debug[0xab55d3]
bin/mysqld-debug[0x606178]
bin/mysqld-debug(_Z24plugin_foreach_with_maskP3THDPFcS0_PP13st_plugin_intPvEijS4_+0x308)[0x7c3d6c]
bin/mysqld-debug(_Z16ha_drop_databasePc+0x2e)[0x60b038]
bin/mysqld-debug(_Z11mysql_rm_dbP3THDPcbb+0x420)[0x778aa4]
bin/mysqld-debug(_Z21mysql_execute_commandP3THD+0x4230)[0x7b7774]
bin/mysqld-debug(_Z11mysql_parseP3THDPcjP12Parser_state+0x41f)[0x7bb5d5]
bin/mysqld-debug(_Z16dispatch_command19enum_server_commandP3THDPcj+0xce4)[0x7bcfb8]
bin/mysqld-debug(_Z10do_commandP3THD+0x309)[0x7be7c1]
bin/mysqld-debug(_Z24do_handle_one_connectionP3THD+0x1ad)[0x77546d]
bin/mysqld-debug(handle_one_connection+0x2d)[0x775533]
bin/mysqld-debug(pfs_spawn_thread+0x18c)[0xd4446a]
/lib64/libpthread.so.0[0x3b25607851]
/lib64/libc.so.6(clone+0x6d)[0x3b24ee811d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fc210004fd0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
[14 Aug 2013 9:26] Sveta Smirnova
I believe workaround here can be to use release version of MySQL server.
[18 Aug 2013 9:25] Justin Swanhart
I was not using a debug mysql!
[18 Sep 2013 19:15] Bugs System
Noted in 5.6.14 changelog:

"An assertion would be raised in "fil_node_open_file" due to a missing
".ibd" file. Instead of asserting, "InnoDB" should return false and the
caller of "fil_node_open_file" should handle the return message."