Bug #80278 Injecting GTID empty transaction is not working with multi source replication
Submitted: 5 Feb 2016 10:51 Modified: 11 Jul 2016 12:08
Reporter: Ramesh Sivaraman Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.7.10 OS:CentOS
Assigned to: CPU Architecture:Any

[5 Feb 2016 10:51] Ramesh Sivaraman
Description:
If any of the replication slave channel is failed in multi source replication setup, injecting GTID empty transaction is not starting the respective slave.

        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
		[..]
                   Last_Errno: 1050
                   Last_Error: Error 'Table 't1' already exists' on query. Default database: 'test'. Query: 'create table t1(id int, primary key(id))'
                [..] 
           Retrieved_Gtid_Set: 064c5d57-cbcc-11e5-b033-002590e9b7a2:1-3
            Executed_Gtid_Set: 026648a1-cbcc-11e5-a077-002590e9b7a2:1,
24a145a6-cbcc-11e5-9a0e-002590e9b7a2:1-2
                Auto_Position: 1
         Replicate_Rewrite_DB: 
                 Channel_Name: master3

mysql> SET GTID_NEXT="064c5d57-cbcc-11e5-b033-002590e9b7a2:3";
Query OK, 0 rows affected (0.00 sec)

mysql> begin;commit;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.09 sec)

mysql> SET GTID_NEXT="AUTOMATIC";
Query OK, 0 rows affected (0.00 sec)

mysql> 
mysql> start slave;

        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
	                      [..]
                   Last_Errno: 1050
                   Last_Error: Error 'Table 't1' already exists' on query. Default database: 'test'. Query: 'create table t1(id int, primary key(id))'
                              [..]
           Retrieved_Gtid_Set: 064c5d57-cbcc-11e5-b033-002590e9b7a2:1-3
            Executed_Gtid_Set: 026648a1-cbcc-11e5-a077-002590e9b7a2:1,
064c5d57-cbcc-11e5-b033-002590e9b7a2:3,
24a145a6-cbcc-11e5-9a0e-002590e9b7a2:1-2
                Auto_Position: 1
         Replicate_Rewrite_DB: 
                 Channel_Name: master3

How to repeat:
execute same DDL/DML from any of two master servers in multi source replication
[15 Feb 2016 23:11] Matthew Lord
Hi Ramesh,

From the output, it seems that you're attempting to skip the GTID "064c5d57-cbcc-11e5-b033-002590e9b7a2:3", assuming that is the one leading to the error? I don't see any GTIDs from 064c5d57-cbcc-11e5-b033-002590e9b7a2 in the GTID_EXECUTED set the first time around, but only in the GTID_RECEIVED set. So we could reasonably assume that it was "064c5d57-cbcc-11e5-b033-002590e9b7a2:1" causing the error, but that should be double checked. If it *is* 064c5d57-cbcc-11e5-b033-002590e9b7a2:1 causing the error, then overwriting 064c5d57-cbcc-11e5-b033-002590e9b7a2:3 with a local empty transaction wouldn't move us forward. 

To verify that this is the correct one to skip, can you share the output of:
1. select * from performance_schema.replication_applier_status_by_worker\G
2. select last_seen_transaction from performance_schema.replication_applier_status_by_worker where channel_name="master3" and last_error_number=1050;

If I missed or misunderstood something, let me know.

Thank you!

Matt
[18 Feb 2016 4:18] Ramesh Sivaraman
Hi Matt,

Yes you are right GTID 064c5d57-cbcc-11e5-b033-002590e9b7a2:1 was causing the error. I was overwriting wrong GTID (GTID 064c5d57-cbcc-11e5-b033-002590e9b7a2:3) with local empty transaction. Sorry for the invalid bug report.

Thanks you very much for the clarification.

Thanks,
Ramesh
[11 Jul 2016 12:08] MySQL Verification Team
Closing the report per last note.