MySQL Bugs: #54009: Server crashes when data is selected from non backed up table for InnoDB plugin

Bug #54009	Server crashes when data is selected from non backed up table for InnoDB plugin
Submitted:	26 May 2010 22:26	Modified:	14 Dec 2010 20:10
Reporter:	Hema Sridharan	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: InnoDB Plugin storage engine	Severity:	S2 (Serious)
Version:	5.1.48	OS:	Linux
Assigned to:	Marko Mäkelä	CPU Architecture:	Any
Tags:	corruption crash

Description:
* Create database dbb and tables dbb.td1, dbb.td2, dbb.td3, dbb.te1, dbb.te2. dbb.tf1
* Dump some data in these tables
* Execute innobackup to backup only tables starting from 'td' using --include regexp
* Apply logs for recovering the tables
* Drop database dbb and shutdown the server
* Perform innobackup --copy-back operation
* Restore the server
* Check that dbb.td* tables are restored and also verify that no other tables are recovered.
* Select data from dbb.td* tables and dbb.te1, dbb.te2 and dbb.tf1
* When data is selected from dbb.te1, MySQL server crashes for InnoDB plugin.

How to repeat:
Please see the detailed test case to reproduce the crash.

==========================================================================

TEST                                      RESULT   TIME (ms)
------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009^M
meb.innobackup_crash                       [ pass ]  36981
meb.innobackup_crash 'innodb_plugin'       [ fail ]
        Test ended at 2010-05-26 23:35:29

CURRENT_TEST: meb.innobackup_crash
mysqltest: At line 93: query 'SELECT COUNT(*) FROM dbb.te1' failed with wrong errno 2013: 'Lost connection to MySQL server during query', instead of 1146..

Here is the crash dump for the test,

100527  0:35:28  InnoDB: Assertion failure in thread 1087293760 in file fil/fil0fil.c line 2981
InnoDB: Failing assertion: flags != DICT_TF_COMPACT
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
100527  0:35:28 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60559 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x12b47588
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x40cec100 thread_stack 0x40000
(my_print_stacktrace+0x32)[0xb05f72]
(handle_segfault+0x28a)[0x698974]
/lib64/libpthread.so.0[0x36a3c0de60]
/lib64/libc.so.6(gsignal+0x35)[0x36a3030045]
/lib64/libc.so.6(abort+0x110)[0x36a3031ae0]
[0x2aaaaac43f5a]
[0x2aaaaac3befa]
[0x2aaaaac323ac]
[0x2aaaaac32d70]
[0x2aaaaac5aad9]
[0x7eb247]
(_Z21open_table_from_shareP3THDP14st_table_sharePKcjjjP8st_tableb+0xbd9)[0x70d05f]
[0x6ff457]
(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootPbj+0xe7b)[0x701fa7]
(_Z11open_tablesP3THDPP10TABLE_LISTPjj+0x3f7)[0x703007]
(_Z28open_and_lock_tables_derivedP3THDP10TABLE_LISTb+0x85)[0x703991]
(_Z20open_and_lock_tablesP3THDP10TABLE_LIST+0x22)[0x6b69b2]
[0x6a91a9]
(_Z21mysql_execute_commandP3THD+0x8cf)[0x6aa073]
(_Z11mysql_parseP3THDPKcjPS2_+0x2a5)[0x6b2b53]
(_Z16dispatch_command19enum_server_commandP3THDPcj+0xbe0)[0x6b4314]
(_Z10do_commandP3THD+0x252)[0x6b5762]
(handle_one_connection+0x13d)[0x6a1ba9]
/lib64/libpthread.so.0[0x36a3c062e7]
/lib64/libc.so.6(clone+0x6d)[0x36a30ce3bd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x12ba3698 = SELECT COUNT(*) FROM dbb.te1
thd->thread_id=1
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

The test file that reproduces crash is attached below

Attachment: innobackup_crash.test (application/test, text), 3.51 KiB.

Thank you for the report.

In my environment mysqld doesn't crash, but test fails with

mysqltest: At line 90: query 'SELECT COUNT(*) FROM dbb.te1' succeeded - should have failed with errno 1146...

both if run with innodb and with innodb_plugin. Please check if this shows problems with test or this is new bug.

Hi Sveta, 

I forgot to mention that multiple table spaces feature has to be enabled when --include option is used with innobackup. Please repeat the test with --innodb-file-per-table option enabled. 

You can create innobackup_crash-master.opt file, set --innodb-file-per-table=1 in it and then execute the test to reproduce the crash.

Thank you for the feedback.

Verified as described.

I understood that MEB is separate from the MySQL server. If a piece of SQL crashes the MySQL server (mysqld), then that should be filed as a mysqld bug. Which MySQL server does this occur with? Were the data files restored from a backup?

This is not a bug in Innobackup or ibbackup. The procedure described above under "Description" is illegal use of Innobackup. So, this could be described as a user error.
I think it is very difficult if not impossible to change Innobackup to perform the operation tried to achieve in this case. Even though a table is missing from a partial backup, the meta data for it is in the data dictionary which is included in any partial backup.

Using partial backups is very dangerous and the users should be very careful with partial backups.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110480

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110481

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110482

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110483

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110488

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110489

Pushed into 5.1.49 (revid:build@mysql.com-20100719143034-omcma40sblwmay3x) (version source revid:vasil.dimov@oracle.com-20100704071244-3lo4okzels3kvy1p) (merge vers: 5.1.49) (pib:16)

Pushed into mysql-trunk 5.5.6-m3 (revid:alik@sun.com-20100723121820-jryu2fuw3pc53q9w) (version source revid:alik@sun.com-20100723121820-jryu2fuw3pc53q9w) (merge vers: 5.5.6-m3) (pib:18)

Pushed into mysql-next-mr (revid:alik@sun.com-20100723121929-90e9zemk3jkr2ocy) (version source revid:alik@sun.com-20100723121827-3bsh51m5sj6g4oma) (pib:18)

Related bugs are:
http://bugs.mysql.com/bug.php?id=54742
http://bugs.mysql.com/bug.php?id=56373

Pushed into mysql-5.1-telco-7.0 5.1.51-ndb-7.0.20 (revid:martin.skold@mysql.com-20101014082627-jrmy9xbfbtrebw3c) (version source revid:martin.skold@mysql.com-20101014082627-jrmy9xbfbtrebw3c) (merge vers: 5.1.51-ndb-7.0.20) (pib:21)

Pushed into mysql-5.1-telco-6.3 5.1.51-ndb-6.3.39 (revid:martin.skold@mysql.com-20101014083757-5qo48b86d69zjvzj) (version source revid:martin.skold@mysql.com-20101014083757-5qo48b86d69zjvzj) (merge vers: 5.1.51-ndb-6.3.39) (pib:21)

Pushed into mysql-5.1-telco-6.2 5.1.51-ndb-6.2.19 (revid:martin.skold@mysql.com-20101014084420-y54ecj85j5we27oa) (version source revid:martin.skold@mysql.com-20101014084420-y54ecj85j5we27oa) (merge vers: 5.1.51-ndb-6.2.19) (pib:21)