Bug #92219 Under RECOVERING Received_transaction_set and GTID_EXECUTED upside down
Submitted: 29 Aug 2018 6:48 Modified: 31 Oct 2018 1:01
Reporter: Shinya Sugiyama Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Group Replication Severity:S3 (Non-critical)
Version:5.7 OS:Any
Assigned to: MySQL Verification Team CPU Architecture:Any
Tags: group replication, InnoDB Cluster

[29 Aug 2018 6:48] Shinya Sugiyama
Description:
Under Recovery Condition, Received_transaction_set and GTID_EXECUTED became upside down.

Under normal condition, everything working fine.

-------------------------------------
 Please confirm following example
-------------------------------------

localhost [(none)]>  SELECT * FROM performance_schema.replication_group_members;SELECT member_host as "primary master" FROM performance_schema.global_status JOIN performance_schema.replication_group_members WHERE variable_name = 'group_replication_primary_member' AND member_id=variable_value;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 42c3cbdc-d216-14h9-b56c-02011477005e | TST-db-12   |        3306 | ONLINE       |
| group_replication_applier | e1731e29-d216-14h9-8646-020114770060 | TST-db-14   |        3306 | RECOVERING   |
| group_replication_applier | e53e17a3-d216-14h9-98c5-02011477005f | TST-db-13   |        3306 | ONLINE       |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)

+----------------+
| primary master |
+----------------+
| TST-db-12      |
+----------------+
1 row in set (0.00 sec)

It seems "Received_transaction_set" is not showing latest status.
It is under "@@global.GTID_EXECUTED".

localhost [performance_schema]> select Received_transaction_set FROM performance_schema.replication_connection_status WHERE Channel_name = 'group_replication_applier'; select @@global.GTID_EXECUTED;
+--------------------------------------------------------------------------------------------+
| Received_transaction_set                                                                   |
+--------------------------------------------------------------------------------------------+
| cb904887-a1fc-11e8-ae58-02011477005e:1-24598537                                            |
+--------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

+--------------------------------------------------------------------------------------------+
| @@global.GTID_EXECUTED                                                                     |
+--------------------------------------------------------------------------------------------+
| cb904887-a1fc-11e8-ae58-02011477005e:1-24598559                                            |
+--------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

localhost [performance_schema]> select @last_exec:=SUBSTRING_INDEX(SUBSTRING_INDEX(SUBSTRING_INDEX( @@global.GTID_EXECUTED,':',-1),':',1),'-',-1) last_executed;select @last_rec:=SUBSTRING_INDEX(SUBSTRING_INDEX(SUBSTRING_INDEX( Received_transaction_set,':',-1),':',1),'-',-1) last_received FROM performance_schema.replication_connection_status WHERE Channel_name = 'group_replication_applier';select (@last_rec - @last_exec) as real_lag;
+---------------+
| last_executed |
+---------------+
| 24598559      |
+---------------+
1 row in set (0.00 sec)

+---------------+
| last_received |
+---------------+
| 24598537      |
+---------------+
1 row in set (0.00 sec)

+----------+
| real_lag |
+----------+
|      -22 | ← "In this case under -22"
+----------+
1 row in set (0.00 sec)

How to repeat:
1) STOP ONE OF INSTANCE UNDER GROUP REPLICATION.
2) RUN Large Volume of Transaction. (Such as delete large table)
3) START STOPPED INSTANCE.
4) THE INSTANCE BECAME RECOVERING MODE
5) Please check the lag between Received_transaction_set and @@global.GTID_EXECUTED.

Suggested fix:
Received_transaction_set should be under @@global.GTID_EXECUTED.
[30 Oct 2018 3:01] MySQL Verification Team
Hi,

I am unable to reproduce this issue following your scenario. Can you reproduce this issue on-demand? If you can, please upload the configs and the log files and the redolog and binlog files from all nodes so we can try to figure out how the issue happened from the logs.

Thanks
Bogdan
[31 Oct 2018 1:01] Shinya Sugiyama
Hi Bogdan,

Thank you for investigating this issue.
Since I already place all of those servers in production, 
I can't do benchmark anymore.

Just in case, I attached an sample option file.
If it can't reproduce error on you end, please close this ticket.

Best Regards
Shinya
[31 Oct 2018 1:04] Shinya Sugiyama
Sample Option File

Attachment: sample_parameter.txt (text/plain), 6.08 KiB.