MySQL Bugs: #99132: Group Replication Relay Node documentation is missing

Bug #99132	Group Replication Relay Node documentation is missing
Submitted:	31 Mar 2020 14:54	Modified:	7 Apr 2020 14:17
Reporter:	Tibor Korocz	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: Documentation	Severity:	S3 (Non-critical)
Version:	8.0.19	OS:	Linux
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
Hi, 

I have a three node InnoDB Cluster. mysql1,mysql2,mysql3

mysql2 is the Primary , mysql1 and mysql3 are the readers.

If we simulate a partial network outage example with iptables:

Running this on mysql3:

mysql3# iptables -A INPUT -s mysql2 \
      -j DROP; iptables -A OUTPUT -s mysql2 -j DROP

mysql3 will still get all the changes made on mysql2 because mysql1 is going act like a relay node and send all the changes to mysql3. You can confirm this even with tcpdump. 

However this behaviour/function is not well documented. I could not find anything mentioning or detailing this in the documentation. 

How to repeat:
create a 3 node InnoDb cluster.
create a table on primary:

CREATE TABLE `lab` ( `id` int NOT NULL AUTO_INCREMENT, `hostname` varchar(20) DEFAULT NULL, `created_at` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `idx_created` (`created_at`) ) ENGINE=InnoDB;

Insert some data in a loop:
while true;do mysql -usbtest -pxxxxx -P3306 -h127.0.0.1 -e "INSERT INTO sysbench.lab (hostname) VALUES ( @@hostname)"; done 2>/dev/null

Cut network between a reader and primary:
mysql3# iptables -A INPUT -s mysql2 \
      -j DROP; iptables -A OUTPUT -s mysql2 -j DROP

If you select data from mysql3, you still can see new rows are inserted into that table, even if it can not connect to primary directly. 

Suggested fix:
Please update the documentation and describe this behaviour.

Hi Tibor,

> mysql3# iptables -A INPUT -s mysql2 -j DROP; iptables -A OUTPUT -s mysql2 -j DROP

I don't consider this a "normal setup" and any behavior GR shows here depends on the number of factors. Mostly on the GCS (corosync/xcom/..). And this behavior can change in the future, especially with introduction of additional GCS's.

Dunno if you agree or not, but that makes sense to me. 

kind regards
bogdan

Hi  Bogdan, 

I am sorry but I disagree. 
It is a very simple and common setup , 3 node GR with a single primary. 
In the test we can see a behaviour which shows us, there is a core functionality of GR or the protocol. Which is not documented anywhere as far as I can see. 

I would love to see how this is exactly working, is there any way to configure this, etc... ?

Agree that is typical configuration. Node1 in DC1-AZ1, node2 in DC1-AZ2, node3 in DC2-AZ1. Network drops between DC1-AZ1 and DC2-AZ1, but not between the other. Above, Tibor simulates this network outage with iptables.

Yes, this is not a bug but a documentation failure. Where is this behavior described?