Description:
Datenodes added to a four node cluster during SINGLE USER MODE are always added to nodegroup 0. This problem does not occur during multi user mode. Nor does this anomolous state affect availability of data to applications. However, attempts to rectify the state leads to multiple datanode crashes in 5.0.18. I believe that in 5.0.22, safeguards are encountered which prevent crashes and therefore prevent node shutdown.
How to repeat:
# cd /var/lib/mysql-cluster
# more config.ini
[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=10MB
IndexMemory=10MB
Diskless=1
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
[TCP DEFAULT]
# Managment Server
[NDB_MGMD]
HostName=127.0.0.1
# Storage Engines
[NDBD]
HostName=127.0.0.1
DataDir=/var/lib/mysql-cluster
[NDBD]
HostName=127.0.0.1
DataDir=/var/lib/mysql-cluster
[NDBD]
HostName=127.0.0.1
DataDir=/var/lib/mysql-cluster
[NDBD]
HostName=127.0.0.1
DataDir=/var/lib/mysql-cluster
# MySQL Clients
# I personally leave this blank to allow rapid changes of the mysql clients;
# you can enter the hostnames of the above two servers here. I suggest you dont.
[MYSQLD]
[MYSQLD]
# /usr/local/mysql/bin/ndb_mgmd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: 127.0.0.1:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 4 node(s)
id=2 @127.0.0.1 (Version: 5.0.18, Nodegroup: 0, Master)
id=3 @127.0.0.1 (Version: 5.0.18, Nodegroup: 0)
id=4 @127.0.0.1 (Version: 5.0.18, Nodegroup: 1)
id=5 @127.0.0.1 (Version: 5.0.18, Nodegroup: 1)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.0.1 (Version: 5.0.18)
[mysqld(API)] 2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
ndb_mgm> 3 STOP
Node 3: Node shutdown initiated
Node 3 has shutdown.
ndb_mgm> Node 3: Node shutdown completed.
ndb_mgm> 5 STOP
Node 5: Node shutdown initiated
Node 5 has shutdown.
Node 5: Node shutdown completed.
ndb_mgm> SHOW
Cluster Configuration
---------------------
[ndbd(NDB)] 4 node(s)
id=2 @127.0.0.1 (Version: 5.0.18, Nodegroup: 0, Master)
id=3 (not connected, accepting connect from 127.0.0.1)
id=4 @127.0.0.1 (Version: 5.0.18, Nodegroup: 1)
id=5 (not connected, accepting connect from 127.0.0.1)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.0.1 (Version: 5.0.18)
[mysqld(API)] 2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
ndb_mgm> ENTER SINGLE USER MODE 6
Single user mode entered
Access is granted for API node 6 only.
ndb_mgm> EXIT
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> SHOW
Connected to Management Server at: 127.0.0.1:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 4 node(s)
id=2 @127.0.0.1 (Version: 5.0.18, single user mode, Nodegroup: 0, Master)
id=3 @127.0.0.1 (Version: 5.0.18, starting, Nodegroup: 0)
id=4 @127.0.0.1 (Version: 5.0.18, single user mode, Nodegroup: 1)
id=5 @127.0.0.1 (Version: 5.0.18, starting, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 (Version: 5.0.18)
[mysqld(API)] 2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
ndb_mgm> EXIT SINGLE USER MODE
Exiting single user mode in progress.
Use ALL STATUS or SHOW to see when single user mode has been exited.
ndb_mgm> SHOW
Cluster Configuration
---------------------
[ndbd(NDB)] 4 node(s)
id=2 @127.0.0.1 (Version: 5.0.18, Nodegroup: 0, Master)
id=3 @127.0.0.1 (Version: 5.0.18, Nodegroup: 0)
id=4 @127.0.0.1 (Version: 5.0.18, Nodegroup: 1)
id=5 @127.0.0.1 (Version: 5.0.18, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.0.1 (Version: 5.0.18)
[mysqld(API)] 2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
ndb_mgm> 5 STOP
Node 5: Node shutdown initiated
Node 5 has shutdown.
ndb_mgm>
ndb_mgm> Node 5: Forced node shutdown completed. Occured during startphase 2. Initiated by signal 0. Caused by error 2339: 'Send signal error(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
Node 3: Forced node shutdown completed. Occured during startphase 2. Initiated by signal 0. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error
ndb_mgm> SHOW
Cluster Configuration
---------------------
[ndbd(NDB)] 4 node(s)
id=2 @127.0.0.1 (Version: 5.0.18, Nodegroup: 0, Master)
id=3 (not connected, accepting connect from 127.0.0.1)
id=4 @127.0.0.1 (Version: 5.0.18, Nodegroup: 1)
id=5 (not connected, accepting connect from 127.0.0.1)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.0.1 (Version: 5.0.18)
[mysqld(API)] 2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
Suggested fix:
It is strongly recommended that datanodes are not added to a cluster while a cluster is in SINGLE USER MODE. Recommended practice is to not modify schema in SINGLE USER MODE while concurrently restoring datanodes to a cluster and therefore this state is unlikely to occur in the wild.
