Bug #91658 Kubernetes&MGR Executing "start group_replication" fails
Submitted: 16 Jul 2018 9:30 Modified: 16 Jul 2018 9:44
Reporter: Jeffry Ye Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Group Replication Severity:S3 (Non-critical)
Version:5.7.22 OS:CentOS
Assigned to: CPU Architecture:Any
Tags: kubernetes, mgr

[16 Jul 2018 9:30] Jeffry Ye
Description:
Kubernetes&&MGR Executing "start group_replication" fails and error log shows: 

2018-07-16T08:26:52.317000Z 81 [Note] Plugin group_replication reported: 'Group communication SSL configuration: group_replication_ssl_mode: "DISABLED"'
2018-07-16T08:26:52.317184Z 81 [Note] Plugin group_replication reported: '[GCS] Added automatically IP ranges 127.0.0.1/8,172.30.75.72/32 to the whitelist'
2018-07-16T08:26:52.320163Z 81 [Note] Plugin group_replication reported: '[GCS] Translated 'mysql1' to 10.254.119.157'
2018-07-16T08:26:52.320237Z 81 [ERROR] Plugin group_replication reported: '[GCS] There is no local IP address matching the one configured for the local node (mysql1:24901).'

How to repeat:
I have use kubernetes & MySQL Group Replication.
Docker MySQL image version:
	mysql:5.7.22
kubernetes version:
	[root@k8s-ceph-master-57-89 mgr]# rpm -qa |grep kubelet
	kubelet-1.9.0-0.x86_64

[root@k8s-ceph-master-57-89 mgr]# kubectl -n wxj get pods -o wide
NAME                      READY     STATUS    RESTARTS   AGE       IP               NODE
mysql1-6756857c67-mzddj   1/1       Running   0          50m       172.30.75.72     k8s-ceph-node-130-141
mysql2-55cd4589cf-qxhzt   1/1       Running   0          1h        172.30.6.8       k8s-ceph-node-15-103
mysql3-59df464ff5-dnmwl   1/1       Running   0          1h        172.30.253.199   k8s-ceph-node-148-186

[root@k8s-ceph-master-57-89 mgr]# kubectl -n wxj get services -o wide
NAME      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE       SELECTOR
mysql1    ClusterIP   10.254.119.157   <none>        3306/TCP,24901/TCP   51m       app=mysql1
mysql2    ClusterIP   10.254.233.35    <none>        3306/TCP,24901/TCP   51m       app=mysql2
mysql3    ClusterIP   10.254.99.70     <none>        3306/TCP,24901/TCP   51m       app=mysql3

Relevant my.cnf :(mysql1,mysql2,mysql3 is kubernetes service name)
  transaction_write_set_extraction=XXHASH64
  loose-group_replication_group_name="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
  loose-group_replication_start_on_boot=off
  loose-group_replication_local_address= "mysql1:24901"
  loose-group_replication_group_seeds= "mysql1:24901,mysql2:24901,mysql3:24901"
  loose-group_replication_bootstrap_group= off

In pod mysql1-6756857c67-mzddj,I can use service name connect mysql:
	root@mysql1-6756857c67-mzddj:/# mysql -h mysql1 -uroot -pyijiupi -P 3306
	mysql: [Warning] Using a password on the command line interface can be insecure.
	Welcome to the MySQL monitor.  Commands end with ; or \g.
	Your MySQL connection id is 331
	Server version: 5.7.22-log MySQL Community Server (GPL)
	Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
	Oracle is a registered trademark of Oracle Corporation and/or its
	affiliates. Other names may be trademarks of their respective
	owners.
	Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
	mysql> 
	
	root@mysql1-6756857c67-mzddj:/# mysql -h mysql2 -uroot -pyijiupi -P 3306
	mysql: [Warning] Using a password on the command line interface can be insecure.
	Welcome to the MySQL monitor.  Commands end with ; or \g.
	Your MySQL connection id is 698
	Server version: 5.7.22-log MySQL Community Server (GPL)
	Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.	
	Oracle is a registered trademark of Oracle Corporation and/or its
	affiliates. Other names may be trademarks of their respective
	owners.	
	Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
	mysql> select @@hostname;
	+-------------------------+
	| @@hostname              |
	+-------------------------+
	| mysql2-55cd4589cf-qxhzt |
	+-------------------------+

What advice do you have to help me solve this problem? Thinks.

Suggested fix:
I can use Kubernetes services name connection MySQL server,but can't use service name   
initialize MySQL group_replication.
[16 Jul 2018 9:31] Jeffry Ye
the kubernetes conf file

Attachment: k8s_mgr_conf.zip (application/x-zip-compressed, text), 5.64 KiB.

[16 Jul 2018 9:44] Jeffry Ye
reopen
[17 Oct 2018 22:24] Dung Nguyen
I got the same issue and get around it by adding rule to /etc/hosts
In k8s, you can use hostAliases in your pod config

    spec:
      hostAliases:
      - ip: "127.0.0.1"
        hostnames:
        - "YOUR_SERVICE_NAME"