Bug #89506 Add the primary key to the partition table leading to SLAVE database crash
Submitted: 2 Feb 2018 3:39 Modified: 2 Mar 2018 7:54
Reporter: wentao liu Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: DDL Severity:S2 (Serious)
Version:5.7.17 OS:Linux
Assigned to: CPU Architecture:Any
Tags: MySQL5.7.17, partition, primary key, Slave crash

[2 Feb 2018 3:39] wentao liu
Description:
mysql version:5.7.17

When adding a primary key to a partition table on the master, the execution on the master is successful, but the slave is crashing and cannot be started

slave error log:

2018-02-02T03:13:35.545799Z 10 [Note] 'CHANGE MASTER TO FOR CHANNEL '' executed'. Previous state master_host='10.10.10.221', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='10.10.10.221', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''.
2018-02-02T03:13:35.558091Z 11 [Note] Slave I/O thread: Start asynchronous replication to master 'rdsRepl@10.10.10.221:3306' in log 'FIRST' at position 4
2018-02-02T03:13:35.558144Z 11 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2018-02-02T03:13:35.559369Z 11 [Note] Slave I/O thread for channel '': connected to master 'rdsRepl@10.10.10.221:3306',replication started in log 'FIRST' at position 4
2018-02-02T03:13:35.568031Z 12 [Warning] Slave SQL for channel '': If a crash happens this configuration does not guarantee that the relay log info will be consistent, Error_code: 0
2018-02-02T03:13:35.568074Z 12 [Note] Slave SQL thread for channel '' initialized, starting replication in log 'FIRST' at position 0, relay log './rds_mysql-relay-bin.000001' position: 4
2018-02-02T03:13:35.604813Z 13 [ERROR] InnoDB: Table `.`.`test`.`test_partition` has a primary key in InnoDB data dictionary, but not in MySQL!
2018-02-02 03:13:35 0x7f0cd52d7700  InnoDB: Assertion failure in thread 139693092861696 in file ha_innopart.cc line 1198
InnoDB: Failing assertion: ref_length
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
03:13:35 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=1
max_threads=1320
thread_count=7
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 709837 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x14742000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f0cd52d6e60 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xef584b]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x7af3e1]
/lib64/libpthread.so.0(+0xf100)[0x7f0f7c752100]
/lib64/libc.so.6(gsignal+0x37)[0x7f0f7b1475d7]
/lib64/libc.so.6(abort+0x148)[0x7f0f7b148cc8]
/usr/sbin/mysqld[0x77fe7a]
/usr/sbin/mysqld(_ZN11ha_innopart4openEPKcij+0x13fb)[0xf4fbab]
/usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcii+0x33)[0x7fce23]
/usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x777)[0xd5fbf7]
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0xe9b)[0xc6c61b]
/usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x612)[0xc73432]
/usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPKcS2_P24st_ha_create_informationP10TABLE_LISTP10Alter_info+0x2b0)[0xd2fa50]
/usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x4f8)[0xe2cdb8]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x5d0)[0xcc25b0]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3b5)[0xcc8975]
/usr/sbin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcm+0x6a3)[0xe6b9e3]
/usr/sbin/mysqld(_Z27slave_worker_exec_job_groupP12Slave_workerP14Relay_log_info+0x187)[0xecb4e7]
/usr/sbin/mysqld(handle_slave_worker+0x383)[0xeae313]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0x1269f04]
/lib64/libpthread.so.0(+0x7dc5)[0x7f0f7c74adc5]
/lib64/libc.so.6(clone+0x6d)[0x7f0f7b207c0d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (147cbcc9): ALTER TABLE `test_partition`
MODIFY COLUMN `ddid`  varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL FIRST ,
MODIFY COLUMN `currenttime`  datetime NOT NULL AFTER `ddid`,
ADD PRIMARY KEY (`currenttime`, `ddid`),
ADD UNIQUE INDEX `currenttime` (`currenttime`) USING BTREE
Connection ID (thread ID): 13
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

How to repeat:
MASTER:create table 
CREATE TABLE `test_partition` (
  `currenttime` datetime NOT NULL,
  `ddid` varchar(200) NOT NULL,
  `publishtime` varchar(128) DEFAULT NULL,
  `link` text,
  `type` text,
  `title` text,
  `taskname` text,
  `isupdate` int(2) NOT NULL DEFAULT '0'
) ENGINE=InnoDB DEFAULT CHARSET=utf8
/*!50100 PARTITION BY RANGE (TO_DAYS(currenttime))
(PARTITION p2016 VALUES LESS THAN (736695) ENGINE = InnoDB,
 PARTITION p2017 VALUES LESS THAN (737060) ENGINE = InnoDB,
 PARTITION p201801 VALUES LESS THAN (737091) ENGINE = InnoDB,
 PARTITION p201802 VALUES LESS THAN (737119) ENGINE = InnoDB,
 PARTITION p201803 VALUES LESS THAN (737150) ENGINE = InnoDB,
 PARTITION p201804 VALUES LESS THAN (737180) ENGINE = InnoDB,
 PARTITION p201805 VALUES LESS THAN (737211) ENGINE = InnoDB,
 PARTITION p201806 VALUES LESS THAN (737241) ENGINE = InnoDB,
 PARTITION p201807 VALUES LESS THAN (737272) ENGINE = InnoDB,
 PARTITION p201808 VALUES LESS THAN (737303) ENGINE = InnoDB,
 PARTITION p201809 VALUES LESS THAN (737333) ENGINE = InnoDB,
 PARTITION p201810 VALUES LESS THAN (737364) ENGINE = InnoDB,
 PARTITION p201811 VALUES LESS THAN (737394) ENGINE = InnoDB,
 PARTITION p201812 VALUES LESS THAN (737425) ENGINE = InnoDB,
 PARTITION p201901 VALUES LESS THAN (737456) ENGINE = InnoDB,
 PARTITION p201902 VALUES LESS THAN (737484) ENGINE = InnoDB,
 PARTITION p201903 VALUES LESS THAN (737515) ENGINE = InnoDB,
 PARTITION p201904 VALUES LESS THAN (737545) ENGINE = InnoDB,
 PARTITION p201905 VALUES LESS THAN (737576) ENGINE = InnoDB,
 PARTITION p201906 VALUES LESS THAN (737606) ENGINE = InnoDB,
 PARTITION p201907 VALUES LESS THAN (737637) ENGINE = InnoDB,
 PARTITION p201908 VALUES LESS THAN (737668) ENGINE = InnoDB,
 PARTITION p201909 VALUES LESS THAN (737698) ENGINE = InnoDB,
 PARTITION p201910 VALUES LESS THAN (737729) ENGINE = InnoDB,
 PARTITION p201911 VALUES LESS THAN (737759) ENGINE = InnoDB,
 PARTITION p201912 VALUES LESS THAN (737790) ENGINE = InnoDB,
 PARTITION p2020 VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */;

insert data:

insert into test_partition select '2017-01-01' ,uuid(),'20170101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-01-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-02-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-03-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-04-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-05-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-06-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-07-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-08-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-09-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-10-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-11-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2018-12-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2019-01-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2019-02-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2019-03-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2019-04-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2019-05-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2019-06-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';
insert into test_partition select '2019-07-01' ,uuid(),'20180101',uuid(),uuid(),uuid(),uuid(),'0';

ALTER TABLE `test_partition`
MODIFY COLUMN `ddid`  varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL FIRST ,
MODIFY COLUMN `currenttime`  datetime NOT NULL AFTER `ddid`,
ADD PRIMARY KEY (`currenttime`, `ddid`),
ADD UNIQUE INDEX `currenttime` (`currenttime`) USING BTREE;

slave:
mysql> desc test_partition;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    12
Current database: test
[2 Feb 2018 7:54] MySQL Verification Team
Hello wentao liu,

Thank you for the report and test case.
I'm not seeing this issue with 5.7.21(even with 5.7.18, I'm unable to locate which exact patch fixed this issue) with the provided test case and with default configuration. Please note that we don't fix bugs in old versions, don't backport bug fixes. So, please upgrade and inform us if the problem still exists in 5.7.21. If you are able to reproduce on latest GAs then provide conf file from both the master and slave to confirm the issue at our end? Thank you.

Thanks,
Umesh
[2 Feb 2018 7:55] MySQL Verification Team
test results

Attachment: 89506.results (application/octet-stream, text), 91.34 KiB.

[3 Mar 2018 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".