Bug #94541 Assertion on import via Transportable Tablespace
Submitted: 4 Mar 2019 9:32 Modified: 4 Sep 2019 14:41
Reporter: Daniël van Eeden (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:5.7.25 OS:Any
Assigned to: CPU Architecture:Any

[4 Mar 2019 9:32] Daniël van Eeden
Description:
2019-03-03 08:02:42 0x7f5454311700  InnoDB: Assertion failure in thread 140000166483712 in file btr0pcur.cc line 454
InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == btr_pcur_get_block(cursor)->page.id.page_no()
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
07:02:42 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=3
max_threads=10000
thread_count=7
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11915972 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f53100249c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f5454310d30 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xef37ab]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x79fb91]
/lib64/libpthread.so.0(+0xf5d0)[0x7f6f285285d0]
/lib64/libc.so.6(gsignal+0x37)[0x7f6f2683f207]
/lib64/libc.so.6(abort+0x148)[0x7f6f268408f8]
/usr/sbin/mysqld[0x76ffda]
/usr/sbin/mysqld[0x10f1bdf]
/usr/sbin/mysqld(_ZN10IndexPurge4nextEv+0x246)[0xff3a16]
/usr/sbin/mysqld(_ZN10IndexPurge15garbage_collectEv+0xba)[0xff3d5a]
/usr/sbin/mysqld(_Z20row_import_for_mysqlP12dict_table_tP14row_prebuilt_t+0x1fbd)[0xff9a0d]
/usr/sbin/mysqld(_ZN11ha_innobase28discard_or_import_tablespaceEc+0x3dd)[0xf3535d]
/usr/sbin/mysqld(_Z34mysql_discard_or_import_tablespaceP3THDP10TABLE_LISTb+0x15c)[0xd2466c]
/usr/sbin/mysqld(_ZN33Sql_cmd_discard_import_tablespace7executeEP3THD+0xdb)[0xe2689b]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x5d0)[0xcb90b0]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3dd)[0xcbf3bd]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xaba)[0xcbff5a]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x19f)[0xcc19ef]
/usr/sbin/mysqld(handle_connection+0x290)[0xd83e30]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0x126b184]
/lib64/libpthread.so.0(+0x7dd5)[0x7f6f28520dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f6f26906ead]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f5310019200): alter table MYTABLE1 import tablespace
Connection ID (thread ID): 8706
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

From c++filt:
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xef37ab]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x79fb91]
/lib64/libpthread.so.0(+0xf5d0)[0x7f6f285285d0]
/lib64/libc.so.6(gsignal+0x37)[0x7f6f2683f207]
/lib64/libc.so.6(abort+0x148)[0x7f6f268408f8]
/usr/sbin/mysqld[0x76ffda]
/usr/sbin/mysqld[0x10f1bdf]
/usr/sbin/mysqld(IndexPurge::next()+0x246)[0xff3a16]
/usr/sbin/mysqld(IndexPurge::garbage_collect()+0xba)[0xff3d5a]
/usr/sbin/mysqld(row_import_for_mysql(dict_table_t*, row_prebuilt_t*)+0x1fbd)[0xff9a0d]
/usr/sbin/mysqld(ha_innobase::discard_or_import_tablespace(char)+0x3dd)[0xf3535d]
/usr/sbin/mysqld(mysql_discard_or_import_tablespace(THD*, TABLE_LIST*, bool)+0x15c)[0xd2466c]
/usr/sbin/mysqld(Sql_cmd_discard_import_tablespace::execute(THD*)+0xdb)[0xe2689b]
/usr/sbin/mysqld(mysql_execute_command(THD*, bool)+0x5d0)[0xcb90b0]
/usr/sbin/mysqld(mysql_parse(THD*, Parser_state*)+0x3dd)[0xcbf3bd]
/usr/sbin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0xaba)[0xcbff5a]
/usr/sbin/mysqld(do_command(THD*)+0x19f)[0xcc19ef]
/usr/sbin/mysqld(handle_connection+0x290)[0xd83e30]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0x126b184]
/lib64/libpthread.so.0(+0x7dd5)[0x7f6f28520dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f6f26906ead]

Anonymized table structure:
Create Table: CREATE TABLE `<hidden>` (
  `<hidden>` bigint(20) unsigned NOT NULL DEFAULT '0',
  `<hidden>` varchar(10) DEFAULT NULL,
  `<hidden>` bigint(20) unsigned NOT NULL DEFAULT '0',
  `<hidden>` text,
  `<hidden>` int(10) unsigned NOT NULL DEFAULT '0',
  `<hidden>` tinyint(4) NOT NULL DEFAULT '0',
  `<hidden>` varchar(255) DEFAULT NULL,
  `<hidden>` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `<hidden>` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `<hidden>` varchar(255) DEFAULT NULL,
  `<hidden>` tinyint(4) NOT NULL DEFAULT '0',
  `<hidden>` double DEFAULT NULL,
  `<hidden>` tinyint(4) NOT NULL DEFAULT '0',
  `<hidden>` tinyint(4) NOT NULL DEFAULT '1',
  `<hidden>` int(10) unsigned DEFAULT NULL,
  `<hidden>` char(3) DEFAULT NULL,
  `<hidden>` float DEFAULT NULL,
  `<hidden>` timestamp NULL DEFAULT NULL,
  `<hidden>` enum('aaa','bbb') DEFAULT 'aaa',
  `<hidden>` float DEFAULT NULL,
  `<hidden>` double DEFAULT NULL,
  `<hidden>` timestamp NULL DEFAULT NULL,
  `<hidden>` tinyint(4) DEFAULT '0',
  `<hidden>` int(11) DEFAULT NULL,
  `<hidden>` int(10) unsigned NOT NULL DEFAULT '0',
  `<hidden>` date DEFAULT NULL,
  `<hidden>` timestamp NULL DEFAULT NULL,
  `<hidden>` varchar(8) DEFAULT NULL,
  `<hidden>` binary(16) DEFAULT NULL COMMENT 'foo bar',
  PRIMARY KEY (`<hidden>`),
  KEY `<hidden>` (`<hidden>`),
  KEY `<hidden>` (`<hidden>`),
  KEY `<hidden>` (`<hidden>`,`<hidden>`),
  KEY `<hidden>` (`<hidden>`),
  KEY `<hidden>` (`<hidden>`,`<hidden>`),
  KEY `<hidden>` (`<hidden>`),
  KEY `<hidden>` (`<hidden>`,`<hidden>`),
  KEY `<hidden>` (`<hidden>`),
  KEY `<hidden>` (`<hidden>`),
  KEY `<hidden>` (`<hidden>`,`<hidden>`),
  KEY `<hidden>` (`<hidden>`,`<hidden>`),
  KEY `<hidden>` (`<hidden>`),
  KEY `<hidden>` (`<hidden>`,`<hidden>`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC
1 row in set (0.00 sec)

Source and target run 5.7.25

How to repeat:
unknown

Suggested fix:
Handle this case without killing the server, just give some information about why the imported data is considered to be invalid.
[6 Mar 2019 14:15] MySQL Verification Team
Hi Daniel,

Thank you for your bug report.

I see exactly where the assertion happens, but I can not verify the bug without repeating it. You see, that assertion can happen due to a large variety of causes, so we can not detect which is the one that happened in your case.

Hence, we will need a feedback from you. While we are waiting on your feedback, I will enquire whether it would be possible to break that assertion into a code that will provide the error message for all possible causes which might cause that assertion. I hope that you will agree with me.

Meanwhile .....

Can you provide us with a table definition and contents (I guess there were no rows) and with a tablespace. You can also provide us with a dump of the tablespace, without uploading a tablespace to our sftp server.

Hence, you can use either to upload what is necessary for us in order to repeat the problem or, if there is not much info, to provide all info in a hidden comment.

To upload table and tablespace, you can use "Files" tab from this interface. If data are too large, you can use sftp command, as described in "Files" tab.

While you are preparing what we need, I shall initiate a discussion on whether it would be possible to handle this case without killing the server, but, instead, to provide a verbose error message on the cause of why  the imported data is considered to be invalid.

I am sure that you understood my basic plan on this problem.
[7 Mar 2019 14:01] MySQL Verification Team
Hi Daniel,

We had a discussion on this topic and concluded that this is a bug that requires changes in our code.

So, right now, we do not need any feedback from you.

Verified as reported.
[4 Sep 2019 14:21] Daniel Price
Posted by developer:
 
commit 1d6bcf7fe93a8d3c669c2237532761acf9b873cd
Author: Sachin Agarwal <sachin.z.agarwal@oracle.com>
Date:   Wed Sep 4 13:05:58 2019 +0530

    Bug#29454828  ASSERTION ON IMPORT VIA TRANSPORTABLE TABLESPACE
    
    Problem:
    If we flush a table for EXPORT which has delete-marked rows.
    During flush, PageConverter read the pages and try to purge
    delete-marked records, only if it is possible to do so without
    re-organising the B+tree otherwise increase m_n_purge_failed counter
    in index stats.
    
    Later during IMPORT tablespace, we check m_n_purge_failed counter
    and purge any delete-marked records that couldn't be purged during
    the page conversion phase.
    
    To purge all delete-marked records, IndexPurge::garbage_collect()
    iterate over all the record, if a record is delete-marked then purge
    that record.
    If any page is corrupted and we position the cursor to that page
    then we hit an assert :
    ut_a(btr_page_get_prev(next_page, mtr) ==
    btr_pcur_get_block(cursor)->page.id.page_no())
    
    Instead of hitting any assert, we should terminate import tablespace
    operation and throw an error.
    
    Fix:
    Converted this assert check to if condition check for import
    operation. If a page is corrupted then we terminate import
    tablespace operation and throw an error.
    
    RB: 22895
    Reviewed by: Mayank Prasad <mayank.prasad@oracle.com>
[4 Sep 2019 14:41] Daniel Price
Posted by developer:
 
Fixed as of the upcoming 5.7.29, 8.0.19 releases, and here's the changelog entry:

A tablespace import operation raised an assertion when the cursor was
positioned on a corrupted page while purging delete-marked records.
Instead of asserting when encountering a corrupted page, the import
operation is now terminated and an error is reported.
[4 Sep 2019 15:47] MySQL Verification Team
Thank you, Daniel ......