MySQL Bugs: #11404: mysqld process does not reconnect to cluster if MGM process moves

Bug #11404	mysqld process does not reconnect to cluster if MGM process moves
Submitted:	16 Jun 2005 22:56	Modified:	21 Jun 2005 1:52
Reporter:	Jonathan Miller	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	5.1.0-wl2325-wl1354-new	OS:	Linux (Linux)
Assigned to:	Assigned Account	CPU Architecture:	Any

Description:
Setting the ndbconnect sting in the my.cnf with two server that it can connect to should allow it to reconnect to the cluster if the ndb_mgmd is started on the other system.

Config.ini
[DB DEFAULT]
NoOfReplicas: 2
IndexMemory: 100M
DataMemory: 300M
BackupMemory: 64M
MaxNoOfConcurrentScans: 100
DataDir: .
FileSystemPath: /home/ndbdev/jmiller/builds/run

[MGM DEFAULT]
PortNumber: 14000
ArbitrationRank: 1
DataDir: .
[ndb_mgmd]
Id: 1
HostName: ndb10

[ndb_mgmd]
Id: 13
HostName: ndb11

[ndbd]
Id: 2
HostName: ndb10

[ndbd]
Id: 3
HostName: ndb11

[ndbd]
Id: 4
HostName: ndb10

[ndbd]
Id: 5
HostName: ndb11

[api]
Id: 6
HostName: ndb10

[mysqld]
Id: 7
HostName: ndb10

[mysqld]
Id: 8
HostName: ndb11

my.cnf
[ndbdev@ndb10 run]$ cat /etc/my.cnf
[mysqld]
server-id=3
master-user=rep
master-connect-retry=1
master-host=ndb08
master-password=test
master-port=3307
skip-slave-start
datadir=/home/ndbdev/jmiller/builds/var
log-warnings
log-error=/home/ndbdev/jmiller/builds/var/ndb10.mysql.com.err
ndbcluster
ndb-connectstring=ndb10:14000,ndb11:14000  # location of MGM node

How to repeat:
[ndbdev@ndb10 run]$ ndb_mgmd -f config.ini
Cluster configuration warning:
  arbitrator with id 1 and db node with id 2 on same host ndb10
  Running arbitrator on the same host as a database node may
  cause complete cluster shutdown in case of host failure.
[ndbdev@ndb10 run]$ ndbd -c $ndbc
[ndbdev@ndb10 run]$ ndbd -c $ndbc
[ndbdev@ndb11 run]$ ndbd -c $ndbc
[ndbdev@ndb11 run]$ ndbd -c $ndbc
[ndbdev@ndb10 run]$ mysql_server start
Starting MySQL                                             [  OK  ]

---------------------------------------------------------------------------
ndb_mgm> show
Connected to Management Server at: ndb10:14000
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=2    @10.100.1.95  (Version: 5.1.0, Nodegroup: 0, Master)
id=3    @10.100.1.96  (Version: 5.1.0, Nodegroup: 0)
id=4    @10.100.1.95  (Version: 5.1.0, Nodegroup: 1)
id=5    @10.100.1.96  (Version: 5.1.0, Nodegroup: 1)

[ndb_mgmd(MGM)] 2 node(s)
id=1 (not connected, accepting connect from ndb10)
id=13   @10.100.1.96  (Version: 5.1.0)

[mysqld(API)]   4 node(s)
id=6    @10.100.1.95  (Version: 5.1.0)
id=7 (not connected, accepting connect from ndb10)
id=8 (not connected, accepting connect from ndb11)
id=9 (not connected, accepting connect from ndb11)

----------------------------------------------------------------------------------------
[ndbdev@ndb10 run]$ ndb_mgm $ndbc
-- NDB Cluster -- Management Client --
ndb_mgm> shutdown
Connected to Management Server at: ndb10:14000
4 NDB Cluster storage node(s) have shutdown.
NDB Cluster management server shutdown.
--------------------------------------------------------------------------------------------
[ndbdev@ndb11 run]$ ndb_mgmd -f config.ini
Cluster configuration warning:
  arbitrator with id 1 and db node with id 2 on same host ndb10
  arbitrator with id 13 and db node with id 3 on same host ndb11
  Running arbitrator on the same host as a database node may
  cause complete cluster shutdown in case of host failure.
[ndbdev@ndb11 run]$ ndbd -c ndb11:14000
[ndbdev@ndb11 run]$ ndbd -c ndb11:14000
[ndbdev@ndb10 run]$ ndbd -c ndb11:14000
[ndbdev@ndb10 run]$ ndbd -c ndb11:14000
--------------------------------------------------------------------------------------------
[ndbdev@ndb10 run]$ vi ../var/ndb10.mysql.com.err
Error in mgm protocol parser. cmd: 'get connection parameter' status=1, curr=(null)
Failed to get dynamic port to connect to: -3
----------------------------------------------------------------------------------------------
[ndbdev@ndb11 run]$ ndb_mgm ndb11:14000
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: ndb11:14000
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=2    @10.100.1.95  (Version: 5.1.0, Nodegroup: 0, Master)
id=3    @10.100.1.96  (Version: 5.1.0, Nodegroup: 0)
id=4    @10.100.1.95  (Version: 5.1.0, Nodegroup: 1)
id=5    @10.100.1.96  (Version: 5.1.0, Nodegroup: 1)

[ndb_mgmd(MGM)] 2 node(s)
id=1 (not connected, accepting connect from ndb10)
id=13   @10.100.1.96  (Version: 5.1.0)

[mysqld(API)]   4 node(s)
id=6 (not connected, accepting connect from ndb10)
id=7 (not connected, accepting connect from ndb10)
id=8 (not connected, accepting connect from ndb11)
id=9 (not connected, accepting connect from ndb11)
------------------------------------------------------------------------------
[ndbdev@ndb10 run]$ mysql_server restart
Shutting down MySQL                                        [  OK  ]
Starting MySQL                                             [  OK  ]
--------------------------------------------------------------------------------
[ndbdev@ndb11 run]$ ndb_mgm ndb11:14000
-- NDB Cluster -- Management Client --
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=2    @10.100.1.95  (Version: 5.1.0, Nodegroup: 0, Master)
id=3    @10.100.1.96  (Version: 5.1.0, Nodegroup: 0)
id=4    @10.100.1.95  (Version: 5.1.0, Nodegroup: 1)
id=5    @10.100.1.96  (Version: 5.1.0, Nodegroup: 1)

[ndb_mgmd(MGM)] 2 node(s)
id=1 (not connected, accepting connect from ndb10)
id=13   @10.100.1.96  (Version: 5.1.0)

[mysqld(API)]   4 node(s)
id=6    @10.100.1.95  (Version: 5.1.0)
id=7 (not connected, accepting connect from ndb10)
id=8 (not connected, accepting connect from ndb11)
id=9 (not connected, accepting connect from ndb11)

Suggested fix:
mysqld should try on both system:ports.

I'll take this one as it looks like my code.

After pulling and rebuilding, this now works for me as well, but I still get the following in the error mysqld error log:
Error in mgm protocol parser. cmd: 'get connection parameter' status=1, curr=(null)
Failed to get dynamic port to connect to: -3

Please do not submit the same bug more than once. An existing
bug report already describes this very problem. Even if you feel
that your issue is somewhat different, the resolution is likely
to be the same. Because of this, we hope you add your comments
to the original bug instead.

Thank you for your interest in MySQL.

Additional info:

In this case, then I believe this is technically a duplicate of BUG#10950.

The situation is slightly different in 4.1, but the same fix should apply to both.