Bug #97450 Got an error reading communication packets MySQL docker overlay network
Submitted: 31 Oct 2019 22:24 Modified: 26 Dec 2019 11:54
Reporter: Jun Wang Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Package Repos Severity:S2 (Serious)
Version:5.7.28 OS:Ubuntu (18.06)
Assigned to: MySQL Verification Team CPU Architecture:x86
Tags: docker overlay network, group replication, master master

[31 Oct 2019 22:24] Jun Wang
Description:
Getting infinite [Note] Got an error reading communication packets

 Running master-master replication configuration on two aws EC T2 micro instances.

EC2 instance A: as docker swarm manager.
EC2 instance B: as docker swarm worker.
Customized Overlay Docker network is configured properly. Ping operations are verified to be successful.
Run master 1 MySQL docker instance on EC2 instance A. Instance uprunning not problem. DB access/read/write no issues. However, in the log few warnings are received and the message "[Note] Got an error reading communication packets " has been written to mysqld.log infinitely.

[Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-10-31T19:35:56.598553Z 0 [Warning] CA certificate ca.pem is self signed.
2019-10-31T19:35:56.639651Z 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=master1-relay-bin' to avoid this problem.
[Warning] Recovery from master pos 4 and file  for channel 'group_replication_recovery'. Previous relay log pos and relay log file had been set to 4, ./master1-relay-bin-group_replication_recovery.000002 respectively.

This is likely impacting the normal operation. We have installed the server and it usually go down in 1 day.

How to repeat:
install docker.io on ubuntu 18.04
docker swarm init
docker network create db-overlay

docker run -d --network db-overlay --name=master$N --hostname=master$N \
        -p $gp_port:33061 -p $sql_port:3306 \
        -v $PWD/$log_dir:/var/log/ \
        -v $PWD/$data_dir:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=mypass \
        mysql/mysql-server:5.7 \
        --server_id=$N \
        --log_bin='mysql-bin-1.log' \
        --enforce_gtid_consistency='ON' \
        --log_slave_updates='ON' \
        --gtid_mode='ON' \
        --enforce_gtid_consistency='ON' \
        --binlog_format='ROW' \
        --binlog_checksum='NONE' \
        --master_info_repository='TABLE' \
        --relay_log_info_repository='TABLE' \
        --plugin_load='group_replication.so' \
        --transaction_write_set_extraction='XXHASH64' \
        --relay_log_recovery='ON' \
        --group_replication_group_name='aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee' \
        --group_replication_start_on_boot='OFF' \
        --group_replication_local-address="master"$N":33061" \
        --group_replication_group_seeds='master1:33061,master2:33061,master3:33062' \
        --loose_group_replication_single_primary_mode='OFF' \
        --loose_group_replication_enforce_update_everywhere_checks='ON' \
        --group_replication_bootstrap_group='OFF'

docker exec -it docker_id bash;
mysql -u root -h localhost -p

CREATE USER usr@'%' IDENTIFIED BY 'passwd'; GRANT REPLICATION SLAVE ON *.* TO usr_rpl@'%';  FLUSH PRIVILEGES;

SET GLOBAL group_replication_bootstrap_group=ON; 
START GROUP_REPLICATION; 
SET GLOBAL group_replication_bootstrap_group=OFF;

error message will appear in /var/log/mysqld.log
[20 Dec 2019 17:51] MySQL Verification Team
Hi,

You are reproducing this on bare metal or on AWS only? I tried this on bare metal and on kvm/qemu (proxmox) but on centos and was not reproducing. So before I go retest this on ubuntu I need to know if you managed to reproduce this on bare metal too or AWS only

best regards
[20 Dec 2019 18:05] Jun Wang
Hi there, I am only able to reproduce it on AWS EC2 instances.
[26 Dec 2019 11:54] MySQL Verification Team
Hi,

I cannot reproduce this neither on bare metal nor on qemu/kvm virtualization on linux (proxmox) so I can only assume this is something with AWS.