Bug #99132 Group Replication Relay Node documentation is missing
Submitted: 31 Mar 2020 14:54 Modified: 7 Apr 2020 14:17
Reporter: Tibor Korocz Email Updates:
Status: Not a Bug Impact on me:
Category:MySQL Server: Documentation Severity:S3 (Non-critical)
Version:8.0.19 OS:Linux
Assigned to: MySQL Verification Team CPU Architecture:Any

[31 Mar 2020 14:54] Tibor Korocz

I have a three node InnoDB Cluster. mysql1,mysql2,mysql3

mysql2 is the Primary , mysql1 and mysql3 are the readers.

If we simulate a partial network outage example with iptables:

Running this on mysql3:

mysql3# iptables -A INPUT -s mysql2 \
      -j DROP; iptables -A OUTPUT -s mysql2 -j DROP

mysql3 will still get all the changes made on mysql2 because mysql1 is going act like a relay node and send all the changes to mysql3. You can confirm this even with tcpdump. 

However this behaviour/function is not well documented. I could not find anything mentioning or detailing this in the documentation. 

How to repeat:
create a 3 node InnoDb cluster.
create a table on primary:

CREATE TABLE `lab` ( `id` int NOT NULL AUTO_INCREMENT, `hostname` varchar(20) DEFAULT NULL, `created_at` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `idx_created` (`created_at`) ) ENGINE=InnoDB;

Insert some data in a loop:
while true;do mysql -usbtest -pxxxxx -P3306 -h127.0.0.1 -e "INSERT INTO sysbench.lab (hostname) VALUES ( @@hostname)"; done 2>/dev/null

Cut network between a reader and primary:
mysql3# iptables -A INPUT -s mysql2 \
      -j DROP; iptables -A OUTPUT -s mysql2 -j DROP

If you select data from mysql3, you still can see new rows are inserted into that table, even if it can not connect to primary directly. 

Suggested fix:
Please update the documentation and describe this behaviour.
[7 Apr 2020 13:25] MySQL Verification Team
Hi Tibor,

> mysql3# iptables -A INPUT -s mysql2 -j DROP; iptables -A OUTPUT -s mysql2 -j DROP

I don't consider this a "normal setup" and any behavior GR shows here depends on the number of factors. Mostly on the GCS (corosync/xcom/..). And this behavior can change in the future, especially with introduction of additional GCS's.

Dunno if you agree or not, but that makes sense to me. 

kind regards
[7 Apr 2020 14:17] Tibor Korocz
Hi  Bogdan, 

I am sorry but I disagree. 
It is a very simple and common setup , 3 node GR with a single primary. 
In the test we can see a behaviour which shows us, there is a core functionality of GR or the protocol. Which is not documented anywhere as far as I can see. 

I would love to see how this is exactly working, is there any way to configure this, etc... ?
[30 Oct 2020 20:22] Matthew Boehm
Agree that is typical configuration. Node1 in DC1-AZ1, node2 in DC1-AZ2, node3 in DC2-AZ1. Network drops between DC1-AZ1 and DC2-AZ1, but not between the other. Above, Tibor simulates this network outage with iptables.

Yes, this is not a bug but a documentation failure. Where is this behavior described?