Bug #86822 MySQL crash InnoDB: Assertion failure .. in file rem0rec.cc line 578
Submitted: 26 Jun 2017 4:06 Modified: 13 Aug 2017 13:40
Reporter: Artyom Konovalenko Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:5.7.18 OS:Oracle Linux (Percona build mysqld 5.7.18-15-log)
Assigned to: CPU Architecture:Any
Tags: _Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm, double free or corruption

[26 Jun 2017 4:06] Artyom Konovalenko
Description:
Recently we have upgraded from mysql 5.6.36 to 5.7.18. 
We use 5.7 transportable InnoDB partitions with "PARTITION BY LIST" partitioning. 
We use Percona MySQL build, but it looks like InnoDB bug, so I submit bug report here.

Since starting 5.7.18 we have had 3 server crashes, all with the same or similar backtraces. There is always _Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm in the backtrace.

Here is the last backtrace:

2017-06-26 01:44:32 0x7fc4c81b3700  InnoDB: Assertion failure in thread 140483147544320 in file rem0rec.cc line 578
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
15:44:32 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=138
max_threads=501
thread_count=74
connection_count=73
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 207249 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fc3e7297560
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fc4c81b2d70 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf0334b]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0x7a0731]
/lib64/libpthread.so.0(+0xf100)[0x7fc632aa7100]
/lib64/libc.so.6(gsignal+0x37)[0x7fc630c315f7]
/lib64/libc.so.6(abort+0x148)[0x7fc630c32ce8]
/usr/sbin/mysqld[0x76e6ea]
/usr/sbin/mysqld(_Z20rec_get_offsets_funcPKhPK12dict_index_tPmmPP16mem_block_info_t+0x56)[0x10af916]
/usr/sbin/mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x22ab)[0x110b0cb]
/usr/sbin/mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x316)[0xff4326]
/usr/sbin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x248)[0x80bc58]
/usr/sbin/mysqld(_ZN7handler16read_range_firstEPK12st_key_rangeS2_bb+0x5c)[0x80c40c]
/usr/sbin/mysqld(_ZN7handler21multi_range_read_nextEPPc+0x99)[0x7ff059]
/usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x5a)[0xe0605a]
/usr/sbin/mysqld[0xc3747a]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x11b)[0xca8d8b]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x267)[0xca1967]
/usr/sbin/mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x17d)[0xd13f5d]
/usr/sbin/mysqld[0x75f7c1]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x482e)[0xcd518e]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x60d)[0xcd84dd]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xaba)[0xcd902a]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1df)[0xcdaabf]
/usr/sbin/mysqld(handle_connection+0x2b8)[0xda2f28]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf22a94]
/lib64/libpthread.so.0(+0x7dc5)[0x7fc632a9fdc5]
/lib64/libc.so.6(clone+0x6d)[0x7fc630cf2c4d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.

How to repeat:
Intermediate bug, no stable way to reproduce.
[26 Jun 2017 10:29] MySQL Verification Team
Thank you for the bug report. Please test with MySQL binaries, also you have not commented if you tried to check some remarks done by the error message: have you checked hardware problems, have you checked corruption data?, also how was the process you applied in the upgrade process?. Try to find a repeatable test case. Thanks.
[28 Jun 2017 2:07] Artyom Konovalenko
Hi.
Thank you for reply.

> Please test with MySQL binaries, 
Ok, I will run MySQL binaries from your site.

> also you have not commented if you tried to check some remarks done by the error message: have you checked hardware problems

This is virtual guest. There were no hardware related issues on the host when the problem appeared

> have you checked corruption data?
Yes. We did full logical dump/restore of the data after first crash (so recreated ibd files) and there were definitely no corruption at the moment of 2nd crash. 

> also how was the process you applied in the upgrade process?. 
We followed Percona recommended upgrade procedure: https://www.percona.com/doc/percona-server/LATEST/upgrading_guide_56_57.html

What we did:
- demoted server to slave
- stopped mysqld
- removed all mysql-server, mysql-client, and shared libs binaries (without deps)
- installed 5.7.18 version of the above
- started the server in standalone mode (slave with stopped replication)
- ran mysql_upgrade script (there were no errors)
- restarted mysqld
- started slave,
- waited for replication to be completed
- promoted server to master

> Try to find a repeatable test case. Thanks.
I'll try, but no guarantee.
[28 Jun 2017 15:33] MySQL Verification Team
Hi!

You need to provide us with additional info. Have your run mysql_upgrade and have you altered your partitions to native  partitions ????

If not, please, follow those steps.
[29 Jun 2017 1:36] Artyom Konovalenko
We had run mysql_upgrade, then we altered original tables (there were no partitions initially). 

It was one big alter, that changed:
- primary key
- added column 
- changed default charset for table (was utf8, new latin1)
- changed block_size (was=default, new=2)
- added native partitions

old definition (before alter):
 CREATE TABLE `table1` (
  `id` bigint(20) NOT NULL,
  `format` varchar(32) NOT NULL,
  `meta` text NOT NULL,
  `created_date` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`format`,`id`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED;

new definition (after alter):
 CREATE TABLE `table1` (
  `id` bigint(20) NOT NULL,
  `format` varchar(32) CHARACTER SET utf8 NOT NULL,
  `meta` text CHARACTER SET utf8 NOT NULL,
  `created_date` timestamp NULL DEFAULT NULL,
  `namespace_id` int(5) NOT NULL DEFAULT '7',
  PRIMARY KEY (`namespace_id`,`format`,`id`),
  KEY `format` (`format`,`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=1
PARTITION BY LIST  COLUMNS(namespace_id)
(PARTITION p_1 VALUES IN (1) ENGINE = InnoDB,
 PARTITION p_2 VALUES IN (2) ENGINE = InnoDB,
 PARTITION p_3 VALUES IN (3) ENGINE = InnoDB,
 PARTITION p_4 VALUES IN (4) ENGINE = InnoDB,
 PARTITION p_5 VALUES IN (5) ENGINE = InnoDB,
 PARTITION p_6 VALUES IN (6) ENGINE = InnoDB,
 PARTITION p_7 VALUES IN (7) ENGINE = InnoDB,
 PARTITION p_8 VALUES IN (8) ENGINE = InnoDB,
 PARTITION p_9 VALUES IN (9) ENGINE = InnoDB)

P.S. I know, the primary key is ugly (contains varchar, too long), but that's all we have.
[11 Jul 2017 15:17] MySQL Verification Team
Hi !

Thank you for your answers. If Innodb crashes, then we must take a close look at the problem. 

In order to repeat the crash, please make a dump of the table, gzip it or bzip it and upload it to this bug, by using "Files" tab.

Thank you very much in advance.
[13 Jul 2017 7:22] Artyom Konovalenko
Hi.

I installed clean mysqld 5.7.18 from RPM from mysql site and made it master (there were recent crash with the similar backtrace on Percona build). 
Now waiting for crash to happen on clean mysqld, then I update the ticket if it happens again (usually it takes 1-2 weeks from crash to crash). Then I provide you more information.

Best regards,
Arty
[13 Jul 2017 13:40] MySQL Verification Team
Hi!

Thank you for your effort to repeat the bug with our binaries.

If you are successful, please upload the dump of the table in pre-ALTER format, by using "Files" tab of this bug.
[14 Aug 2017 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".