Bug #95292 slave crash when update a table with 'binlog_format=mix' in master
Submitted: 8 May 2019 10:59 Modified: 9 May 2019 15:52
Reporter: li pickup (OCA) Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Row Based Replication ( RBR ) Severity:S2 (Serious)
Version:5.6.33 OS:Red Hat (rhel 7.5)
Assigned to: MySQL Verification Team CPU Architecture:x86 (Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz * )
Tags: crash, RBR, replication, slave

[8 May 2019 10:59] li pickup
Description:
update a table with binlog_format='mix' in the master will cause slave crash.

How to repeat:
* setup a replication between two mysql with my.cnf like 
```
[root@localhost ~]# cat oldmy.cnf
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html
# *** DO NOT EDIT THIS FILE. It's a template which will be copied to the
# *** default location during install, and will be replaced if you
# *** upgrade to a newer version of MySQL.

[mysqld]

# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M

# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin

# These are commonly set, remove the # and set as required.
# basedir = .....
# datadir = .....
# port = .....
# server_id = .....
# socket = .....

# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

log-bin=mysql-bin
server-id=3
binlog-ignore-db=information_schema
binlog-ignore-db=cluster
binlog-ignore-db=mysql
replicate-do-db=liaobei
replicate-ignore-db=mysql
log-slave-updates
slave-skip-errors=all
slave-net-timeout=60

```

* create two table on the master like 
```
CREATE TABLE `chat_userinfo` (
  `id` int(11) NOT NULL AUTO_INCREMENT ,
  `user_id` int(11) DEFAULT NULL ,
  `user_uid` char(40) DEFAULT NULL ,
  `user_uid_parent` char(40) DEFAULT NULL ,
  `user_logo` varchar(255) DEFAULT NULL ,
  `user_name` varchar(100) DEFAULT NULL ,
  `user_sex` tinyint(4) DEFAULT '3' ,
  `user_phone` char(15) DEFAULT NULL ,
  `user_birthday` int(11) DEFAULT NULL ,
  `user_profession` char(50) DEFAULT NULL ,
  `user_describe` char(50) DEFAULT NULL ,
  `user_tag` char(50) DEFAULT NULL ,
  `user_province` varchar(50) DEFAULT NULL,
  `user_address` varchar(255) DEFAULT NULL ,
  `user_gold` decimal(11,2) DEFAULT '0.00' ,
  `user_flag` tinyint(4) DEFAULT '1' ,
  `user_flag_switch` tinyint(4) DEFAULT '1' ,
  `user_auth` tinyint(4) DEFAULT NULL ,
  `user_call_time` int(11) DEFAULT '0' ,
  `user_sort` int(11) DEFAULT '0' ,
  `state` tinyint(4) DEFAULT '1' ,
  `fans_num` int(11) DEFAULT '0' ,
  `call_num` int(11) DEFAULT '0' ,
  `call_through_num` int(11) DEFAULT '0' ,
  `is_recommend` tinyint(4) DEFAULT '1' ,
  `is_active` tinyint(4) DEFAULT '1' ,
  `is_hot` tinyint(4) DEFAULT '1' ,
  `user_city` varchar(50) DEFAULT NULL,
  `user_vip` int(2) DEFAULT '0' ,
  `block_num` int(11) DEFAULT '0' ,
  `hang_num` int(11) DEFAULT '0' ,
  `total_online_time` int(11) DEFAULT '0' ,
  `user_type` tinyint(1) DEFAULT '1' ,
  `channel_id` int(11) DEFAULT '0' ,
  `labor_id` int(11) DEFAULT '0' ,
  `app_flag` char(10) DEFAULT NULL ,
  `ftime` int(11) DEFAULT NULL ,
  `ctime` int(11) DEFAULT NULL ,
  `is_lock` int(11) DEFAULT '1' ,
  `ltime` int(11) DEFAULT NULL ,
  `is_update` tinyint(4) DEFAULT '0' ,
  `ratio` tinyint(3) NOT NULL DEFAULT '60' ,
  `collect_web_socket` tinyint(4) NOT NULL DEFAULT '0',
  `updatetime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `livestate` tinyint(4) NOT NULL,
  PRIMARY KEY (`id`),
  KEY `user_uid` (`user_uid`) USING BTREE,
  KEY `user_flag` (`user_flag`),
  KEY `app_flag` (`app_flag`),
  KEY `user_id` (`user_id`),
  KEY `user_phone` (`user_phone`),
  KEY `ctime` (`ctime`),
  KEY `state` (`state`)
) ENGINE=InnoDB

CREATE TABLE `chat_useraccount` (
  `id` int(11) NOT NULL AUTO_INCREMENT ,
  `user_uid` char(40) DEFAULT '' ,
  `account` varchar(40) DEFAULT '' ,
  `password` varchar(100) DEFAULT NULL ,
  `style` tinyint(11) DEFAULT '1' ,
  `loginIP` varchar(255) DEFAULT NULL ,
  `imei` varchar(50) DEFAULT NULL ,
  `country` varchar(50) DEFAULT NULL ,
  `is_login` int(11) DEFAULT '1' ,
  `ltime` int(11) DEFAULT NULL ,
  `ctime` int(11) DEFAULT NULL ,
  `app_flag` char(10) DEFAULT NULL ,
  `login_time` int(11) DEFAULT NULL ,
  `logout_time` int(11) NOT NULL DEFAULT '0' ,
  `qq_account` varchar(255) DEFAULT NULL,
  `raw_password` varchar(255) DEFAULT NULL,
  `wx_account` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `user_uid` (`user_uid`,`account`,`style`,`app_flag`) USING BTREE,
  KEY `user_uid2` (`user_uid`) USING BTREE,
  KEY `account` (`account`) USING BTREE,
  KEY `style` (`style`) USING BTREE,
  KEY `login_time` (`login_time`)
) ENGINE=InnoDB

```

* update on the master like this
```
update chat_userinfo as info join chat_useraccount as account on info.user_uid = account.user_uid set info.state = 4 where info.user_flag = 2 and account.login_time < unix_timestamp(20190401000000);
```

* the update will log to binlog as binlog_format='Row', and update about 8 rows.

* mysql slave will crash and report error like this
```
08:16:25 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=2
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68108 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f8364000990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f837ed28ea0 thread_stack 0x40000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x911685]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x3d8)[0x675a88]
/lib64/libpthread.so.0(+0xf6d0)[0x7f83a7fd36d0]
/usr/local/mysql/bin/mysqld(_ZNK7handler22ha_statistic_incrementEM17system_status_vary+0x12)[0x58c8a2]
/usr/local/mysql/bin/mysqld(_ZN7handler16ha_external_lockEP3THDi+0x2f)[0x58e0cf]
/usr/local/mysql/bin/mysqld(_Z17mysql_lock_tablesP3THDPP5TABLEjj+0x197)[0x80d7f7]
/usr/local/mysql/bin/mysqld(_Z11lock_tablesP3THDP10TABLE_LISTjj+0x1c7)[0x6a7717]
/usr/local/mysql/bin/mysqld(_Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy+0xa2)[0x6b0f12]
/usr/local/mysql/bin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x521)[0x8a5f21]
/usr/local/mysql/bin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x6d)[0x89f61d]
/usr/local/mysql/bin/mysqld(_Z26apply_event_and_update_posPP9Log_eventP3THDP14Relay_log_info+0x17b)[0x8e579b]
/usr/local/mysql/bin/mysqld[0x8e6195]
/usr/local/mysql/bin/mysqld(handle_slave_sql+0xc85)[0x8e7995]
2019-04-29 15:16:25 18392 [Note] Event Scheduler: Loaded 0 events
2019-04-29 15:16:25 18392 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.6.33-log'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)
/usr/local/mysql/bin/mysqld(pfs_spawn_thread+0x126)[0xae47e6]
/lib64/libpthread.so.0(+0x7e25)[0x7f83a7fcbe25]
/lib64/libc.so.6(clone+0x6d)[0x7f83a6c90bad]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
190429 15:16:25 mysqld_safe Number of processes running now: 0
190429 15:16:25 mysqld_safe mysqld restarted

```
[9 May 2019 15:52] MySQL Verification Team
Hi,

I cannot reproduce this problem on any modern release of mysql.
In addition, please note that "MIXED" is a correct option for binlog format and not "MIX".

thanks