MySQL Bugs: #97694: MySQL 8.0.18 fails on STOP SLAVE/START SLAVE stress test

Bug #97694	MySQL 8.0.18 fails on STOP SLAVE/START SLAVE stress test
Submitted:	19 Nov 2019 15:53	Modified:	22 Jun 2021 20:27
Reporter:	Przemysław Skibiński (OCA)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Replication	Severity:	S3 (Non-critical)
Version:	8.0.18	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
MySQL 8.0.18 fails on STOP SLAVE/START SLAVE stress test.

How to repeat:
Please use attached files:
mysql-test/suite/rpl_gtid/t/rpl_mts_dependency_start_stop_slave_stress.test
mysql-test/suite/rpl_gtid/t/rpl_parallel_simple_load.test
mysql-test/suite/rpl_gtid/t/rpl_parallel_simple_load_cleanup.test
mysql-test/suite/rpl_gtid/t/rpl_parallel_simple_load_consistency.test

Files are taken from https://github.com/facebook/mysql-5.6

Suggested fix:
diff --git a/sql/log_event.cc b/sql/log_event.cc
index 03d10a725b5..4e5e59679ad 100644
--- a/sql/log_event.cc
+++ b/sql/log_event.cc
@@ -5557,6 +5557,16 @@ int Rotate_log_event::do_update_pos(Relay_log_info *rli) {
         routine is being called here.
       */
       if ((error = mts_checkpoint_routine(rli, false))) goto err;
+
+      /*
+        It should be checked after calling mts_checkpoint_routine(), because that
+        function could be interrupted by kill while 'force' is true.
+      */
+      if (sql_slave_killed(thd, rli)) {
+       LogErr(INFORMATION_LEVEL, ER_RPL_SLAVE_ERROR_READING_RELAY_LOG_EVENTS,
+               rli->get_for_channel_str(), "slave SQL thread was killed");
+        goto err;
+      }
     }
 
     mysql_mutex_lock(&rli->data_lock);

MTR test

Attachment: rpl_parallel_simple_load.test (application/octet-stream, text), 2.47 KiB.

MTR test

Attachment: rpl_parallel_simple_load_cleanup.test (application/octet-stream, text), 243 bytes.

MTR test

Attachment: rpl_parallel_simple_load_consistency.test (application/octet-stream, text), 440 bytes.

MTR test

Attachment: rpl_mts_dependency_start_stop_slave_stress.test (application/octet-stream, text), 1.10 KiB.

Error is:

CURRENT_TEST: rpl_gtid.rpl_mts_dependency_start_stop_slave_stress
mysqltest: At line 29: Query 'START SLAVE' failed.
ERROR 1201 (HY000): Could not initialize master info structure; more error messages can be found in the MySQL error log
safe_process[28488]: Child process: 28489, exit: 1

mysqld.2.err contains:

2019-11-19T10:30:19.599959Z 10 [Note] [MY-010572] [Repl] Slave: MTS group recovery relay log info based on Worker-Id 0, group_relay_log_name ./slave-relay-bin.000002, group_relay_log_pos 23527864 group_master_log_name master-bin.000001, group_master_log_pos 23527648
2019-11-19T10:30:19.600125Z 10 [ERROR] [MY-010575] [Repl] Error looking for file after ./slave-relay-bin.000006.

Hello Przemysław,

Thank you for the report and feedback.

Thanks,
Umesh

Posted by developer:
 
Changelog entry added for MySQL 8.0.26:

On a multithreaded replica (with slave_parallel_workers > 0 ), the algorithm used by GTID auto-positioning to check for missing transactions briefly sets a low value (4) for the event position in the course of its calculations. If the operation is stopped at that moment, the recovery process that resolves gaps in the sequence of transactions can fail. The process to resolve gaps is not actually necessary when GTID auto-positioning is used, so the process has now been disabled in that situation.

As a result, on a multithreaded replica, when GTID_MODE = ON is set for the instance and SOURCE_AUTO_POSITION is set for the channel using the CHANGE REPLICATION SOURCE TO statement, the following behaviors now apply: 

A START REPLICA UNTIL SQL_AFTER_MTS_GAPS statement just stops the applier thread when it finds the first event to execute, and does not attempt to check for gaps in the sequence of transactions.

A CHANGE REPLICATION SOURCE TO statement does not automatically fail if there are gaps in the sequence of transactions.

These changed behaviors only apply on a multithreaded replica that uses GTIDs and GTID auto-positioning, and not on a replica that uses binary log position-based replication.

Documentation changes:

https://dev.mysql.com/doc/refman/8.0/en/replication-features-transaction-inconsistencies.h...

When GTID-based replication is in use (GTID_MODE=ON) and SOURCE_AUTO_POSITION is set for the replication channel using the CHANGE REPLICATION SOURCE TO statement, the old relay logs are not required for the recovery process. Instead, the replica can use GTID auto-positioning to calculate what transactions it is missing compared to the source. From MySQL 8.0.26, the process used for binary log position based replication to resolve gaps on a multithreaded replica is skipped entirely when GTID-based replication is in use. When the process is skipped, a START REPLICA UNTIL SQL_AFTER_MTS_GAPS statement behaves differently, and does not attempt to check for gaps in the sequence of transactions. You can also issue CHANGE REPLICATION SOURCE TO statements, which are not permitted on a non-GTID replica where there are gaps. 

https://dev.mysql.com/doc/refman/8.0/en/start-replica.html

From MySQL 8.0.26, the process of checking for gaps in the sequence of transactions is skipped entirely when GTID-based replication and GTID auto-positioning are in use, because gaps in transactions can be resolved using GTID auto-positioning. In that situation, START REPLICA | SLAVE UNTIL SQL_AFTER_MTS_GAPS just stops the applier thread when it finds the first event to execute, and does not attempt to check for gaps in the sequence of transactions. 

https://dev.mysql.com/doc/refman/8.0/en/change-replication-source-to.html

From MySQL 8.0.26, the process of checking for gaps in the sequence of transactions is skipped entirely when GTID-based replication and GTID auto-positioning are in use, because gaps in transactions can be resolved using GTID auto-positioning. In that situation, CHANGE REPLICATION SOURCE TO can still be used.

relay_log_info_repository did not have original behavior documented, not adding as this system variable is deprecated.