Bug #47004 Unable to start the "ndb_mgmd".
Submitted: 31 Aug 2009 8:22 Modified: 3 Oct 2009 7:27
Reporter: Manoj Burande Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:Distrib 5.1.34-ndb-7.0.6 OS:Linux (Fedora 10)
Assigned to: CPU Architecture:Any

[31 Aug 2009 8:22] Manoj Burande
Description:
Hi There,

I just trying to configure the two mgmt nodes on two different host but there is some error while starting the ndb_"mgmd".

Required information is as follows:

Hosts:
192.168.1.107
192.168.1.125

Required configuration for 192.168.1.107:
-----------------------------------------
On 192.168.1.107 : 2 mgmt node(having an entry of another host's mgmt node and itself one).
On 192.168.1.107 : 2 mgmt node(having an entry of another host's mgmt node and itself one).
On 192.168.1.107 : 2 data nodes(having an entry of another host's data node and itself one).

Required configuration for 192.168.1.125:
-----------------------------------------
As same as required by the host 192.168.1.107

The "config.ini" on host:192.168.1.107 configuration file is as,
================================================================
[NDBD DEFAULT]
NoOfReplicas=2    # Number of replicas
DataMemory=3500M    # How much memory to allocate for data storage
IndexMemory=1000M   # How much memory to allocate for index storage
#MaxNoOfConcurrentOperations=1048576
#MaxNoOfConcurrentTransactions= 1048576
#MaxNoOfLocalOperations=1048576
#MaxNoOfConcurrentIndexOperations=16384
#MaxNoOfConcurrentScans=500
#LockPagesInMainMemory=Y
# TCP/IP options:
[TCP DEFAULT]
portnumber=2202   # This the default; however, you can use any

# Management process options:
[NDB_MGMD]
id=1
hostname=192.168.1.107          # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster  # Directory for MGM node logfiles

[NDB_MGMD]
id=2
hostname=192.168.1.125          # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster  # Directory for MGM node logfiles

# Options for data node "A":
[NDBD]
                                # (one [NDBD] section per data node)
id=3
hostname=192.168.1.107          # Hostname or IP address
datadir=/usr/local/mysql/data   # Directory for this data node's datafiles

# Options for data node "B":
[NDBD]
id=4
hostname=192.168.1.125          # Hostname or IP address
datadir=/usr/local/mysql/data   # Directory for this data node's datafiles

# SQL node options:
[MYSQLD]
id=5
hostname=192.168.1.107
[MYSQLD]
id=6
hostname=192.168.1.125

The "my.cnf" on host:192.168.1.107 configuration file is as,
============================================================
# The MySQL server
[mysql_cluster]
ndb-connectstring=nodeid=1,192.168.1.107:1186,192.168.1.125:1186
[ndbd]
connect-string=192.168.1.107:1186,192.168.1.125:1186
[ndb_mgm]
connect-string=nodeid=1,192.168.1.107:1186,192.168.1.125:1186
[ndb_mgmd]
config-file=/var/lib/mysql-cluster/config.ini
[mysqld]
ndbcluster
ndb-connectstring=nodeid=1,192.168.1.107:1186,192.168.1.125:1186

The "config.ini" on host:192.168.1.125 configuration file is as,
================================================================
[NDBD DEFAULT]
NoOfReplicas=2    # Number of replicas
DataMemory=3500M    # How much memory to allocate for data storage
IndexMemory=1000M   # How much memory to allocate for index storage
#MaxNoOfConcurrentOperations=1048576
#MaxNoOfConcurrentTransactions= 1048576
#MaxNoOfLocalOperations=1048576
#MaxNoOfConcurrentIndexOperations=16384
#MaxNoOfConcurrentScans=500
#LockPagesInMainMemory=Y
# TCP/IP options:
[TCP DEFAULT]
portnumber=2202   # This the default; however, you can use any

# Management process options:
[NDB_MGMD]
id=1
hostname=192.168.1.107          # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster  # Directory for MGM node logfiles

[NDB_MGMD]
id=2
hostname=192.168.1.125          # Hostname or IP address of MGM node
datadir=/var/lib/mysql-cluster  # Directory for MGM node logfiles

# Options for data node "A":
[NDBD]
                                # (one [NDBD] section per data node)
id=3
hostname=192.168.1.107          # Hostname or IP address
datadir=/usr/local/mysql/data   # Directory for this data node's datafiles

# Options for data node "B":
[NDBD]
id=4
hostname=192.168.1.125          # Hostname or IP address
datadir=/usr/local/mysql/data   # Directory for this data node's datafiles

# SQL node options:
[MYSQLD]
id=5
hostname=192.168.1.107

The "my.cnf" on host:192.168.1.125 configuration file is as,
============================================================
# The MySQL server
[mysql_cluster]
ndb-connectstring=nodeid=2,192.168.1.107:1186,192.168.1.125:1186
[ndbd]
connect-string=nodeid=192.168.1.107:1186,192.168.1.125:1186
[ndb_mgm]
connect-string=nodeid=2,192.168.1.107:1186,192.168.1.125:1186
[ndb_mgmd]
config-file=/var/lib/mysql-cluster/config.ini
[mysqld]
ndbcluster
ndb-connectstring=nodeid=2,192.168.1.107:1186,192.168.1.125:1186

After all configuration done (on both server) I am trying to start the mgmt using the following command as(on both server),

[root@test mysql-cluster]# ndb_mgmd --initial --connect-string=nodeid=1,192.168.1.125:1186,192.168.1.107:1186
2009-08-31 13:02:41 [MgmSrvr] INFO     -- NDB Cluster Management Server. mysql-5.1.34 ndb-7.0.6
2009-08-31 13:02:42 [MgmSrvr] INFO     -- Reading cluster configuration from '/var/lib/mysql-cluster/config.ini'
2009-08-31 13:02:42 [MgmSrvr] WARNING  -- at line 44: Cluster configuration warning:
  arbitrator with id 1 and db node with id 3 on same host 192.168.1.107
  arbitrator with id 2 and db node with id 4 on same host 192.168.1.125
  Running arbitrator on the same host as a database node may
  cause complete cluster shutdown in case of host failure.

After that I did on "host:192.168.1.107" the following :
------------------------------------------------------
[root@ns mysql-cluster]#  ndb_mgm -e show
Connected to Management Server at: 192.168.1.107:1186
ERROR Message: The cluster configuration is not yet confirmed by all defined management servers. This management server is still waiting for node 2 to connect.

Could not get configuration
*  4012: Failed to get configuration
*        The cluster configuration is not yet confirmed by all defined management servers. This management server is still waiting for node 2 to connect.

and on "host:192.168.1.125":
--------------------------
[root@test mysql-cluster]# ndb_mgm -e show
Connected to Management Server at: 192.168.1.125:1186
ERROR Message: The cluster configuration is not yet confirmed by all defined management servers. This management server is still waiting for node 1 to connect.

Could not get configuration
*  4012: Failed to get configuration
*        The cluster configuration is not yet confirmed by all defined management servers. This management server is still waiting for node 1 to connect.
                                                                                                                                          
                                                                       
Please help me. I am working on the same since 2 days but the proper solution is not getting from everywhere. I am stucked on this issue.

So please provide me the proper steps to configure multiple management nodes with respect to the above mentioned scenario.
 

How to repeat:
1] [root@test mysql-cluster]# ndb_mgmd --connect-string=nodeid=1,192.168.1.107:1186,192.168.1.125:1186

2] [root@ns mysql-cluster]#  ndb_mgm -e show

OR 
3] [root@test mysql-cluster]# ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: 192.168.1.107:1186
ERROR Message: The cluster configuration is not yet confirmed by all defined management servers. This management server is still waiting for node 2 to connect.

Could not get configuration
*  4012: Failed to get configuration
*        The cluster configuration is not yet confirmed by all defined management servers. This management server is still waiting for node 2 to connect.
[31 Aug 2009 9:38] Sveta Smirnova
Thank you for the report.

Have you started both ndb_mgmd nodes? Can they communicate to each other?
[31 Aug 2009 10:02] Manoj Burande
Hello Sveta Smirnova,

I have started both ndb_mgmd nodes as you can see it in my report. After starting the ndb_mgmd it show me "Connected to Management Server at: respective server(192.168.1.107/192.168.1.125):1186. But whenever I tries to the command "ndb_mgm -e show" it reports me the same error as,

ERROR Message: The cluster configuration is not yet confirmed by all defined management
servers. This management server is still waiting for node 2 to connect.

Could not get configuration
*  4012: Failed to get configuration
*        The cluster configuration is not yet confirmed by all defined management servers.
This management server is still waiting for node 2 to connect.
[31 Aug 2009 10:42] Sveta Smirnova
Thank you for the feedback.

Your configuration for host 192.168.1.107 has

[MYSQLD]
id=6
hostname=192.168.1.125

While configuration for host 192.168.1.125 has not such a record.

So this is expected Cluster rejects to start.

See http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-limitations-multiple-nodes.html for details
[31 Aug 2009 11:09] Manoj Burande
Dear Sveta Smirnova,

That was my mistake, sorry for the same..!!! Previously I was forgotten to add these entries in previous description. But the configuration of "config.ini" is identical on the both server. 

Still I am facing the same error. Please help........

For your information:

From the Host:192.168.1.107
===========================
[root@ns mysql-cluster]# netstat -pan | grep EST
tcp        0      0 192.168.1.107:46580         192.168.1.125:22            ESTABLISHED 5414/ssh            
tcp        0      0 192.168.1.107:1186          192.168.1.125:43378         ESTABLISHED 10284/ndb_mgmd      
tcp        0      0 127.0.0.1:1186              127.0.0.1:60277             ESTABLISHED 10284/ndb_mgmd      
tcp        0      0 127.0.0.1:60277             127.0.0.1:1186              ESTABLISHED 10284/ndb_mgmd      
tcp        0      0 ::ffff:192.168.1.107:44904  ::ffff:192.168.1.98:5222    ESTABLISHED 3768/java   

From the Host:192.168.1.125
===========================
[root@test mysql-cluster]# netstat -pan | grep EST
tcp        0      0 192.168.1.125:22            192.168.1.107:46580         ESTABLISHED 2384/0              
tcp        0      0 127.0.0.1:38739             127.0.0.1:1186              ESTABLISHED 3017/ndb_mgmd       
tcp        0      0 192.168.1.125:43378         192.168.1.107:1186          ESTABLISHED 3017/ndb_mgmd       
tcp        0      0 127.0.0.1:1186              127.0.0.1:38739             ESTABLISHED 3017/ndb_mgmd       

The above "netstat" result shows the both management servers have established the connection to each other.

NOTE: Or better way can you please provide me the steps to configure multiple "ndb_mgm". So I will reconfigure it.
[3 Sep 2009 7:27] Hartmut Holzgraefe
Your netstat output shows a mix of IPv4 and IPv6 addresses,
several IPv6 related problems were fixed in MySQL Cluster 7.0.7 so you might want to retry with that version
[3 Oct 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".