Bug #89123 There is no local IP address matching the one configured for the local node
Submitted: 5 Jan 2018 19:58 Modified: 13 Aug 2018 13:00
Reporter: MATTHEW MASSA Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Group Replication Severity:S3 (Non-critical)
Version:5.7.20, 5.7.21, 5.7.22 OS:MacOS (10.11.6 (El Capitan))
Assigned to: CPU Architecture:Any

[5 Jan 2018 19:58] MATTHEW MASSA
Description:
Executing "start group_replication" fails and error log shows: 

'[GCS] There is no local IP address matching the one configured for the local node (192.168.34.11:33061).'

How to repeat:
It seems like the plugin doesn't read the bond0 interface IP address as it should:

ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	nd6 options=1<PERFORMNUD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 5000
	options=10b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV>
	ether a8:20:66:51:24:bf 
	media: autoselect (1000baseT <full-duplex>)
	status: active
en1: flags=8823<UP,BROADCAST,SMART,SIMPLEX,MULTICAST> mtu 1500
	ether 88:53:95:2a:6c:b9 
	nd6 options=1<PERFORMNUD>
	media: autoselect (<unknown type>)
	status: inactive
en2: flags=963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX> mtu 1500
	options=60<TSO4,TSO6>
	ether 32:00:13:47:98:60 
	media: autoselect <full-duplex>
	status: inactive
fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 4078
	lladdr 10:dd:b1:ff:fe:34:79:86 
	nd6 options=1<PERFORMNUD>
	media: autoselect <full-duplex>
	status: inactive
en4: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 5000
	options=10b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV>
	ether a8:20:66:51:24:bf 
	media: autoselect (1000baseT <full-duplex>)
	status: active
p2p0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 2304
	ether 0a:53:95:2a:6c:b9 
	media: autoselect
	status: inactive
awdl0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> mtu 1484
	ether 3e:f5:e3:03:31:be 
	nd6 options=1<PERFORMNUD>
	media: autoselect
	status: inactive
bond0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 5000
	options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
	ether a8:20:66:51:24:bf 
	inet6 fe80::aa20:66ff:fe51:24bf%bond0 prefixlen 64 scopeid 0xb 
	inet 192.168.34.11 netmask 0xffffff00 broadcast 192.168.34.255
	nd6 options=1<PERFORMNUD>
	media: autoselect (1000baseT <full-duplex>)
	status: active
	bond interfaces: en0 en4
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=63<RXCSUM,TXCSUM,TSO4,TSO6>
	ether aa:20:66:15:fd:00 
	Configuration:
		id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
		maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
		root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
		ipfilter disabled flags 0x2
	member: en2 flags=3<LEARNING,DISCOVER>
	        ifmaxaddr 0 port 6 priority 0 path cost 0
	nd6 options=1<PERFORMNUD>
	media: <unknown type>
	status: inactive
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 5000
	options=3<RXCSUM,TXCSUM>
	ether a8:20:66:51:24:bf 
	inet6 fe80::aa20:66ff:fe51:24bf%vlan0 prefixlen 64 scopeid 0xd 
	inet 10.100.6.11 netmask 0xffffff00 broadcast 10.100.6.255
	nd6 options=1<PERFORMNUD>
	vlan: 6 parent interface: bond0
	media: autoselect (1000baseT <full-duplex>)
	status: active

Relevant my.cnf:
# Shared replication group configuration
loose-group_replication_group_name = "A6D308C2-320E-4F10-9E04-2873F7D91931"
loose-group_replication_ip_whitelist = "192.168.34.11/24,192.168.34.110/24"
loose-group_replication_group_seeds = "192.168.34.11:33061,192.168.34.110:33061"

# Single or Multi-primary mode? Uncomment these two lines
# for multi-primary mode, where any host can accept writes
loose-group_replication_single_primary_mode = OFF
loose-group_replication_enforce_update_everywhere_checks = ON

# Host specific replication configuration
server_id = 11
bind-address = "192.168.34.11"
report_host = "192.168.34.11"
loose-group_replication_local_address = "192.168.34.11:33061"

Mysql error log:

2018-01-05T19:32:36.858291Z 14 [Note] Plugin group_replication reported: 'Group communication SSL configuration: group_replication_ssl_mode: "REQUIRED"; server_key_file: "server-key.pem"; server_cert_file: "server-cert.pem"; client_key_file: "server-key.pem"; client_cert_file: "server-cert.pem"; ca_file: "ca.pem"; ca_path: ""; cipher: ""; tls_version: "TLSv1,TLSv1.1"; crl_file: ""; crl_path: ""'
2018-01-05T19:32:36.858804Z 14 [Note] Plugin group_replication reported: '[GCS] Unable to probe network interface "en0" for IP and netmask information. Skipping!'
2018-01-05T19:32:36.858838Z 14 [Note] Plugin group_replication reported: '[GCS] Unable to probe network interface "en2" for IP and netmask information. Skipping!'
2018-01-05T19:32:36.858859Z 14 [Note] Plugin group_replication reported: '[GCS] Unable to probe network interface "fw0" for IP and netmask information. Skipping!'
2018-01-05T19:32:36.858878Z 14 [Note] Plugin group_replication reported: '[GCS] Unable to probe network interface "en4" for IP and netmask information. Skipping!'
2018-01-05T19:32:36.858917Z 14 [ERROR] Plugin group_replication reported: '[GCS] There is no local IP address matching the one configured for the local node (192.168.34.11:33061).'
2018-01-05T19:32:36.858945Z 14 [ERROR] Plugin group_replication reported: 'Unable to initialize the group communication engine'
2018-01-05T19:32:36.858960Z 14 [ERROR] Plugin group_replication reported: 'Error on group communication engine initialization'
2018-01-05T19:32:36.858970Z 14 [Note] Plugin group_replication reported: 'Requesting to leave the group despite of not being a member'
2018-01-05T19:32:36.858982Z 14 [ERROR] Plugin group_replication reported: 'Error calling group communication interfaces while trying to leave the group'

Suggested fix:
When this is done on an Xserve with the same configuration, it works properly.  Affected server (mac mini)  has a LAG interface just like the Xserve but one of the LAG interfaces is a Thunderbolt Gigabit Ethernet adapter (since a Mac Mini doesn't have 2 Gig interfaces).  Not sure how the group_replication plugin is trying to read the IP information, but the ifconfig output from a server that replication startup actually works on for the bond0 interface is identical:

ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	nd6 options=1<PERFORMNUD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 5000
	options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
	ether 00:24:36:f3:00:a2 
	media: autoselect (1000baseT <full-duplex>)
	status: active
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 5000
	options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
	ether 00:24:36:f3:00:a2 
	media: autoselect (1000baseT <full-duplex>)
	status: active
fw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 4078
	lladdr 00:23:df:ff:fe:dd:31:66 
	nd6 options=1<PERFORMNUD>
	media: autoselect <full-duplex>
	status: inactive
bond0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 5000
	options=b<RXCSUM,TXCSUM,VLAN_HWTAGGING>
	ether 00:24:36:f3:00:a2 
	inet6 fe80::224:36ff:fef3:a2%bond0 prefixlen 64 scopeid 0x7 
	inet 192.168.34.110 netmask 0xffffff00 broadcast 192.168.34.255
	nd6 options=1<PERFORMNUD>
	media: autoselect (1000baseT <full-duplex>)
	status: active
	bond interfaces: en0 en1
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 5000
	options=3<RXCSUM,TXCSUM>
	ether 00:24:36:f3:00:a2 
	inet6 fe80::224:36ff:fef3:a2%vlan0 prefixlen 64 scopeid 0x8 
	inet 10.100.6.110 netmask 0xffffff00 broadcast 10.100.6.255
	nd6 options=1<PERFORMNUD>
	vlan: 6 parent interface: bond0
	media: autoselect (1000baseT <full-duplex>)
	status: active
[8 Jan 2018 6:50] MySQL Verification Team
Hello MATTHEW,

Thank you for the report.
Discussed internally with Dev and concluded that it is duplicate of Bug #86772 and per bug updates this has been fixed in upcoming versions and the following was added to the 5.7.21  / 8.0.4 changelog:
Regardless of the number of virtual IPs configured on a machine, Group Replication could access only the first 12 addresses.

Thanks,
Umesh
[8 Jan 2018 14:38] MATTHEW MASSA
Did you notice that the IP in question in my case is the only IP address on the interface?  That doesn’t seem to fit the same conditions as the other bug...
[8 Jan 2018 16:50] MATTHEW MASSA
For example, when I parse for IPv4 addresses on this server there are literally only 3:

ifconfig | grep "inet "
	inet 127.0.0.1 netmask 0xff000000 
	inet 192.168.34.11 netmask 0xffffff00 broadcast 192.168.34.255
	inet 10.100.6.11 netmask 0xffffff00 broadcast 10.100.6.255

That comes nowhere near the limit of 12 on an interface and these are on 3 different interfaces.  Can you please check one more time?
[9 Jan 2018 4:17] MySQL Verification Team
Thank you for the feedback. Please note that even if you feel that your issue is somewhat different but the fix for other issue also fixes this. I'll give it a try on both the versions(reported and fixed) and would let you know. In the meantime please provide conf files used in your environment to try at our end.

Thank you for your interest in MySQL.

Thanks,
Umesh
[17 Jan 2018 11:27] MySQL Verification Team
Also, 5.7.21 is released, may I request you to give it a try and confirm if you are still seeing the issue? I'm not seeing this issue at my end.

Regards,
Umesh
[28 Jan 2018 3:57] MATTHEW MASSA
Gave it a shot tonight, unfortunately it looks like the new version won't run on El Capitan (10.11) only 10.12 and 10.13?  Have a version that will run on 10.11 by any chance?
[12 Feb 2018 11:10] Pratap Pradhan
Hi Umesh,

I have tested the same using MySQL 5.7.21 but i am getting same issues.

2018-02-12T10:52:48.139181Z 11 [Note] Plugin group_replication reported: '[GCS] Translated 'mysql1' to x.x.x.x'
2018-02-12T10:52:48.139203Z 11 [ERROR] Plugin group_replication reported: '[GCS] There is no local IP address matching the one configured for the local node (mysql1:13306).'
2018-02-12T10:52:48.139216Z 11 [ERROR] Plugin group_replication reported: 'Unable to initialize the group communication engine'
2018-02-12T10:52:48.139221Z 11 [ERROR] Plugin group_replication reported: 'Error on group communication engine initialization'
2018-02-12T10:52:48.139225Z 11 [Note] Plugin group_replication reported: 'Requesting to leave the group despite of not being a member'
2018-02-12T10:52:48.139228Z 11 [ERROR] Plugin group_replication reported: 'Error calling group communication interfaces while trying to leave the group'
2018-02-12T10:57:49.227068Z 7 [Note] Aborted connection 7 to db: 'unconnected' user: 'root' host: 'mysql1' (Got timeout reading communication packets)

Please update on same.
[20 Apr 2018 21:07] David Ko
I'm also seeing this on a 5.7.21 instance installed on FreeBSD 11.

Sample log from mysql.err:

2018-04-20T10:09:09.828533Z 23 [Note] Plugin group_replication reported: '[GCS] Unable to probe network interface "igb0" for IP and netmask information. Skipping!'
2018-04-20T10:09:09.828572Z 23 [Note] Plugin group_replication reported: '[GCS] Unable to probe network interface "igb1" for IP and netmask information. Skipping!'
2018-04-20T10:09:09.828597Z 23 [ERROR] Plugin group_replication reported: '[GCS] There is no local IP address matching the one configured for the local node (xx.xx.xx.xx:33061).'

Network configuration:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 
        hwaddr 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 
        hwaddr 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 
        inet xx.xx.xx.xx netmask 0xffffe000 broadcast xx.xx.xx.xx
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: lagg
        laggproto lacp lagghash l2,l3,l4
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

MySQL version:

# mysql --version
mysql  Ver 14.14 Distrib 5.7.21, for FreeBSD11.1 (amd64) using  EditLine wrapper
[24 Apr 2018 16:00] Tiago Vale
Posted by developer:
 
Hi everyone,

Thank you for your interest in MySQL, and for your input on this bug in
particular.
I am just leaving this message here to assure you we are working on this. :-)
[30 May 2018 15:44] MATTHEW MASSA
I was able to test this again with the latest release 5.7.22 (cluster 7.5.10 release) and we're still seeing it.
[13 Aug 2018 13:00] David Moss
Posted by developer:
 
Thank you for your feedback, this has been fixed in upcoming versions and the following was added to the 5.7.23 changelog:
This bug was fixed by backporting the fix for bug#27376511.