MySQL Bugs: #92106: Master is killing the zombie dump thread notices

Bug #92106	Master is killing the zombie dump thread notices
Submitted:	21 Aug 2018 17:26	Modified:	16 Oct 2018 10:24
Reporter:	Ed Dawley	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Documentation	Severity:	S3 (Non-critical)
Version:	5.7	OS:	Any
Assigned to:		CPU Architecture:	Any
Tags:	note, replication, upgrade

Description:
When upgrading a slave from MySQL 5.6 to MySQL 5.7 with default values for slave_net_timeout and MASTER_HEARTBEAT_PERIOD, the slave's heartbeat value will remain at 18400 while the slave_net_timeout will drop to 60.  This effectively disables the heartbeat and causes the master to spa the error log with the following notice:

While initializing dump thread for slave with UUID <d42e1622-510a-11e7-a8a7-528083f2a46f>, found a zombie dump thread with the same UUID. Master is killing the zombie dump thread(20).

There are numerous reported bugs here (#79771, #84475) and posts on stackoverflow that are ultimately caused by this issue.  It would be nice if the upgrade documentation mentioned this as a possible upgrade issue.  

If nothing else, this bug will hopefully show up in Google and save someone else some time.

How to repeat:
1.  Setup replication on a 5.6 slave with default slave_net_timeout and MASTER_HEARTBEAT_PERIOD
2.  Upgrade to 5.7
3.  Start slave
4.  Watch the master's error log.

Hi,
Yes, it would not be bad if we document this a bit better, thanks for your report. I believe our dev team will improve documentation.

kind regards
Bogdan

Posted by developer:

Thanks for reporting this issue. I have investigated the background and will update the documentation soon for this and some related fixes.

Issue analysis:

On a MySQL 5.6 server slave_net_timeout defaults to 3600 seconds (1 hour). If the default is in use, the heartbeat interval therefore defaults to half this period, calculated as 1800 seconds (half an hour). The heartbeat interval can be set using the MASTER_HEARTBEAT_PERIOD option for the CHANGE MASTER TO statement. In MySQL 5.6, omitting MASTER_HEARTBEAT_PERIOD from CHANGE MASTER TO caused the heartbeat interval to be reset to the default (this behavior was stopped in MySQL 5.7 by Bug#18185490). So a 5.6 slave with a default slave_net_timeout is quite likely to have the heartbeat interval as the default 1800 seconds.

The heartbeat interval is recorded with the other master connection information in the slave's master info log (the mysql.slave_master_info table or master.info file). When the slave is upgraded to MySQL 5.7, the new slave_net_timeout default value of 60 seconds is applied. However, the upgrade does not cause the heartbeat interval recorded in the master info log to be recalculated. If the default of 1800 seconds was in place for 5.6, the heartbeat signal is never going to be sent to the slave before the new connection timeout interval expires. If the level of activity on the master is such that updates to the binary log are sent to the slave at least once every 60 seconds, this is not an issue. However, if no data is received from the master, because the heartbeat is not being sent, the connection timeout expires. The slave therefore thinks the connection to the master has been lost and makes multiple reconnection attempts (as controlled by the MASTER_CONNECT_RETRY and MASTER_RETRY_COUNT settings, which can also be seen in the master info log). The reconnection attempts spawn numerous zombie dump threads that the master must kill, causing the master's error log to contain multiple instances of the error ER_RPL_ZOMBIE_ENCOUNTERED.

To avoid this issue, immediately before upgrading a replication slave to MySQL 5.7, check whether slave_net_timeout is using the default setting. If so, issue CHANGE MASTER TO with the MASTER_HEARTBEAT_PERIOD option, and set the heartbeat interval to 30 seconds, so that it will work with the new connection timeout interval of 60 seconds that will apply after the upgrade.

Posted by developer:
 
Documentation updates made in the following sections:

https://dev.mysql.com/doc/refman/5.7/en/upgrading-from-previous-series.html
https://dev.mysql.com/doc/refman/5.7/en/replication-options-slave.html#option_mysqld_maste...
https://dev.mysql.com/doc/refman/5.7/en/replication-options-slave.html#option_mysqld_slave...
https://dev.mysql.com/doc/refman/5.7/en/change-master-to.html