MySQL Bugs: #33789: 2 mgm servers connected but not visible in "show" command after downgrade

Bug #33789	2 mgm servers connected but not visible in "show" command after downgrade
Submitted:	10 Jan 2008 11:01	Modified:	8 Apr 2008 8:29
Reporter:	Lars Torstensson	Email Updates:
Status:	Unsupported	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S3 (Non-critical)
Version:	ndb-6.2.4	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
Having a 4 node cluster with 2 mgm servers. The 2 mgm servers are connected to the data nodes according to the cluster log, but they fail to be visible in the "show" command. You can actually restart a mgm node that isn't connected from one that is (restart 63, from 62).

To get them to talk to each other we had to telnet to each of the mgm servers, after that they registered with each other(or caused by)...  

/hss/i686/6.0.0/apps/database/bin> getdbconnectstring
172.24.0.31:21131,172.24.0.32:21132

/hss/i686/6.0.0/apps/database/bin> ndb_mgm --ndb-connectstring=172.24.0.32:21132
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: 172.24.0.32:21132
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=1    @172.24.0.7  (mysql-5.1.19 ndb-6.2.4, not started)
id=2    @172.24.0.13  (mysql-5.1.19 ndb-6.2.4, not started)
id=3    @172.24.0.7  (mysql-5.1.19 ndb-6.2.4, not started)
id=4    @172.24.0.13  (mysql-5.1.19 ndb-6.2.4, not started)

[ndb_mgmd(MGM)] 2 node(s)
id=62 (not connected, accepting connect from hss-dbmgmd-1)
id=63   (mysql-5.1.19 ndb-6.2.4)

/hss/i686/6.0.0/apps/database/bin> ndb_mgm --ndb-connectstring=172.24.0.31:21131
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: 172.24.0.31:21131
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=1    @172.24.0.7  (mysql-5.1.19 ndb-6.2.4, not started)
id=2    @172.24.0.13  (mysql-5.1.19 ndb-6.2.4, not started)
id=3    @172.24.0.7  (mysql-5.1.19 ndb-6.2.4, not started)
id=4    @172.24.0.13  (mysql-5.1.19 ndb-6.2.4, not started)

[ndb_mgmd(MGM)] 2 node(s)
id=62   (mysql-5.1.19 ndb-6.2.4)
id=63 (not connected, accepting connect from hss-dbmgmd-2) 

How to repeat:
This is the tricky part.

start with having 4 datanodes and 2 mgm servers using 6.2.4

1) 
6.2.4 mgm1    6.2.4 mgm2

2) upgrade to 6.2.9 

3) mgm1 and mgm2 connects and registers ok

4) 6.2.9 mgm1   6.2.9 mgm2 

5) downgrade mgm1 to 6.2.4

6) mgm1 and mgm2 will not register with each other using show.

7) downgrade mgm2 to 6.24 

8) mgm1 and mgm2 will still not register with each other using show.

9) restart whole cluster and  mgm1 and mgm2 will still not register with each other using show.

10) telnet to each of the mgm servers to check connectivity, after that they registered with each other(or caused by it)..

CD

Attachment: mgmd2_cluster.rar (application/octet-stream, text), 3.77 KiB.

This is more or less expected to give problems. The versions used 6.2.4 and 6.2.9 are not really downward compatible, due to e.g. the MicroGCP protocol change in 6.2.5.

We need to do a better job in documenting downgrading, just like we do with upward compatibility with the normal MySQL 5.1. This is still on the TODO.

I'll put the bug report as unsupported. The documenting will be handled eventually.