Bug #91658 Kubernetes&MGR Executing "start group_replication" fails

Submitted: 16 Jul 2018 9:30 Modified: 16 Jul 2018 9:44
Reporter: Jeffry Ye Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Group Replication Severity:S3 (Non-critical)
Version:5.7.22 OS:CentOS
Assigned to: CPU Architecture:Any
Tags: kubernetes, mgr

[16 Jul 2018 9:30] Jeffry Ye
Description:
Kubernetes&&MGR Executing "start group_replication" fails and error log shows: 

2018-07-16T08:26:52.317000Z 81 [Note] Plugin group_replication reported: 'Group communication SSL configuration: group_replication_ssl_mode: "DISABLED"'
2018-07-16T08:26:52.317184Z 81 [Note] Plugin group_replication reported: '[GCS] Added automatically IP ranges 127.0.0.1/8,172.30.75.72/32 to the whitelist'
2018-07-16T08:26:52.320163Z 81 [Note] Plugin group_replication reported: '[GCS] Translated 'mysql1' to 10.254.119.157'
2018-07-16T08:26:52.320237Z 81 [ERROR] Plugin group_replication reported: '[GCS] There is no local IP address matching the one configured for the local node (mysql1:24901).'

How to repeat:
I have use kubernetes & MySQL Group Replication.
Docker MySQL image version:
	mysql:5.7.22
kubernetes version:
	[root@k8s-ceph-master-57-89 mgr]# rpm -qa |grep kubelet
	kubelet-1.9.0-0.x86_64

[root@k8s-ceph-master-57-89 mgr]# kubectl -n wxj get pods -o wide
NAME                      READY     STATUS    RESTARTS   AGE       IP               NODE
mysql1-6756857c67-mzddj   1/1       Running   0          50m       172.30.75.72     k8s-ceph-node-130-141
mysql2-55cd4589cf-qxhzt   1/1       Running   0          1h        172.30.6.8       k8s-ceph-node-15-103
mysql3-59df464ff5-dnmwl   1/1       Running   0          1h        172.30.253.199   k8s-ceph-node-148-186

[root@k8s-ceph-master-57-89 mgr]# kubectl -n wxj get services -o wide
NAME      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE       SELECTOR
mysql1    ClusterIP   10.254.119.157   <none>        3306/TCP,24901/TCP   51m       app=mysql1
mysql2    ClusterIP   10.254.233.35    <none>        3306/TCP,24901/TCP   51m       app=mysql2
mysql3    ClusterIP   10.254.99.70     <none>        3306/TCP,24901/TCP   51m       app=mysql3

Relevant my.cnf :(mysql1,mysql2,mysql3 is kubernetes service name)
  transaction_write_set_extraction=XXHASH64
  loose-group_replication_group_name="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
  loose-group_replication_start_on_boot=off
  loose-group_replication_local_address= "mysql1:24901"
  loose-group_replication_group_seeds= "mysql1:24901,mysql2:24901,mysql3:24901"
  loose-group_replication_bootstrap_group= off

In pod mysql1-6756857c67-mzddj,I can use service name connect mysql:
	root@mysql1-6756857c67-mzddj:/# mysql -h mysql1 -uroot -pyijiupi -P 3306
	mysql: [Warning] Using a password on the command line interface can be insecure.
	Welcome to the MySQL monitor.  Commands end with ; or \g.
	Your MySQL connection id is 331
	Server version: 5.7.22-log MySQL Community Server (GPL)
	Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
	Oracle is a registered trademark of Oracle Corporation and/or its
	affiliates. Other names may be trademarks of their respective
	owners.
	Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
	mysql> 
	
	root@mysql1-6756857c67-mzddj:/# mysql -h mysql2 -uroot -pyijiupi -P 3306
	mysql: [Warning] Using a password on the command line interface can be insecure.
	Welcome to the MySQL monitor.  Commands end with ; or \g.
	Your MySQL connection id is 698
	Server version: 5.7.22-log MySQL Community Server (GPL)
	Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.	
	Oracle is a registered trademark of Oracle Corporation and/or its
	affiliates. Other names may be trademarks of their respective
	owners.	
	Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
	mysql> select @@hostname;
	+-------------------------+
	| @@hostname              |
	+-------------------------+
	| mysql2-55cd4589cf-qxhzt |
	+-------------------------+

What advice do you have to help me solve this problem? Thinks.

Suggested fix:
I can use Kubernetes services name connection MySQL server,but can't use service name   
initialize MySQL group_replication.
[16 Jul 2018 9:31] Jeffry Ye
the kubernetes conf file

Attachment: k8s_mgr_conf.zip (application/x-zip-compressed, text), 5.64 KiB.

[16 Jul 2018 9:44] Jeffry Ye
reopen
[17 Oct 2018 22:24] Dung Nguyen
I got the same issue and get around it by adding rule to /etc/hosts
In k8s, you can use hostAliases in your pod config

    spec:
      hostAliases:
      - ip: "127.0.0.1"
        hostnames:
        - "YOUR_SERVICE_NAME"

Portions of this website are copyright © 2001, 2002 The PHP Group

Page generated in 0.061 sec. using MySQL 8.4.7-u6-cloud

Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the respective copyright holders.
It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.