Bug #20395 Cluster Datanodes Always Assigned To NodeGroup 0 During SINGLE USER MODE
Submitted: 12 Jun 2006 13:37 Modified: 17 Nov 2006 15:53
Reporter: Dean Swift Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:5.0.18 and 5.0.22 OS:Linux (SUSE10)
Assigned to: david li CPU Architecture:Any

[12 Jun 2006 13:37] Dean Swift
Description:
Datenodes added to a four node cluster during SINGLE USER MODE are always added to nodegroup 0. This problem does not occur during multi user mode. Nor does this anomolous state affect availability of data to applications. However, attempts to rectify the state leads to multiple datanode crashes in 5.0.18. I believe that in 5.0.22, safeguards are encountered which prevent crashes and therefore prevent node shutdown.

How to repeat:
# cd /var/lib/mysql-cluster
# more config.ini
[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=10MB
IndexMemory=10MB
Diskless=1

[MYSQLD DEFAULT]

[NDB_MGMD DEFAULT]

[TCP DEFAULT]

# Managment Server

[NDB_MGMD]
HostName=127.0.0.1

# Storage Engines

[NDBD]
HostName=127.0.0.1
DataDir=/var/lib/mysql-cluster

[NDBD]
HostName=127.0.0.1
DataDir=/var/lib/mysql-cluster

[NDBD]
HostName=127.0.0.1
DataDir=/var/lib/mysql-cluster

[NDBD]
HostName=127.0.0.1
DataDir=/var/lib/mysql-cluster

# MySQL Clients
# I personally leave this blank to allow rapid changes of the mysql clients;
# you can enter the hostnames of the above two servers here. I suggest you dont.

[MYSQLD]

[MYSQLD]

# /usr/local/mysql/bin/ndb_mgmd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: 127.0.0.1:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=2    @127.0.0.1  (Version: 5.0.18, Nodegroup: 0, Master)
id=3    @127.0.0.1  (Version: 5.0.18, Nodegroup: 0)
id=4    @127.0.0.1  (Version: 5.0.18, Nodegroup: 1)
id=5    @127.0.0.1  (Version: 5.0.18, Nodegroup: 1)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @127.0.0.1  (Version: 5.0.18)

[mysqld(API)]   2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)

ndb_mgm> 3 STOP
Node 3: Node shutdown initiated
Node 3 has shutdown.

ndb_mgm> Node 3: Node shutdown completed.

ndb_mgm> 5 STOP
Node 5: Node shutdown initiated
Node 5 has shutdown.

Node 5: Node shutdown completed.
ndb_mgm> SHOW
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=2    @127.0.0.1  (Version: 5.0.18, Nodegroup: 0, Master)
id=3 (not connected, accepting connect from 127.0.0.1)
id=4    @127.0.0.1  (Version: 5.0.18, Nodegroup: 1)
id=5 (not connected, accepting connect from 127.0.0.1)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @127.0.0.1  (Version: 5.0.18)

[mysqld(API)]   2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)

ndb_mgm> ENTER SINGLE USER MODE 6
Single user mode entered
Access is granted for API node 6 only.
ndb_mgm> EXIT
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndbd
# /usr/local/mysql/bin/ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> SHOW
Connected to Management Server at: 127.0.0.1:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=2    @127.0.0.1  (Version: 5.0.18, single user mode, Nodegroup: 0, Master)
id=3    @127.0.0.1  (Version: 5.0.18, starting, Nodegroup: 0)
id=4    @127.0.0.1  (Version: 5.0.18, single user mode, Nodegroup: 1)
id=5    @127.0.0.1  (Version: 5.0.18, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1   (Version: 5.0.18)

[mysqld(API)]   2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)

ndb_mgm> EXIT SINGLE USER MODE
Exiting single user mode in progress.
Use ALL STATUS or SHOW to see when single user mode has been exited.
ndb_mgm> SHOW
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=2    @127.0.0.1  (Version: 5.0.18, Nodegroup: 0, Master)
id=3    @127.0.0.1  (Version: 5.0.18, Nodegroup: 0)
id=4    @127.0.0.1  (Version: 5.0.18, Nodegroup: 1)
id=5    @127.0.0.1  (Version: 5.0.18, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @127.0.0.1  (Version: 5.0.18)

[mysqld(API)]   2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)

ndb_mgm> 5 STOP
Node 5: Node shutdown initiated

Node 5 has shutdown.

ndb_mgm>
ndb_mgm> Node 5: Forced node shutdown completed. Occured during startphase 2. Initiated by signal 0. Caused by error 2339: 'Send signal error(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
Node 3: Forced node shutdown completed. Occured during startphase 2. Initiated by signal 0. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error

ndb_mgm> SHOW
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=2    @127.0.0.1  (Version: 5.0.18, Nodegroup: 0, Master)
id=3 (not connected, accepting connect from 127.0.0.1)
id=4    @127.0.0.1  (Version: 5.0.18, Nodegroup: 1)
id=5 (not connected, accepting connect from 127.0.0.1)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @127.0.0.1  (Version: 5.0.18)

[mysqld(API)]   2 node(s)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)

Suggested fix:
It is strongly recommended that datanodes are not added to a cluster while a cluster is in SINGLE USER MODE. Recommended practice is to not modify schema in SINGLE USER MODE while concurrently restoring datanodes to a cluster and therefore this state is unlikely to occur in the wild.
[7 Sep 2006 5:57] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/11517

ChangeSet@1.2255, 2006-09-07 13:58:09+08:00, dli@dev3-76.dev.cn.tlan +4 -0
  Fix for Bug #20395 Cluster datanodes always assigned to nodegroup 0 during single user mode.
  Data node is not allowed to get added in the cluster when it is in single user mode.
[11 Sep 2006 3:01] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/11669

ChangeSet@1.2255, 2006-09-11 11:03:14+08:00, dli@dev3-76.dev.cn.tlan +5 -0
  Fix for Bug #20395 Cluster datanodes always assigned to nodegroup 0 during single user mode.
  Data node is not allowed to get added in the cluster when it is in single user mode.
[10 Oct 2006 18:30] Jonas Oreland
pushed into 5.1.12
[1 Nov 2006 14:37] Jonas Oreland
pushed into 5.0.29
[16 Nov 2006 14:39] Jon Stephens
Changed category to Server:Cluster.
[17 Nov 2006 15:53] Jon Stephens
Thank you for your bug report. This issue has been committed to our source repository of that product and will be incorporated into the next release.

If necessary, you can access the source repository and build the latest available version, including the bug fix. More information about accessing the source trees is available at

    http://dev.mysql.com/doc/en/installing-source.html

Documented bugfix for 5.0.30 and 5.1.12.

Added warning to ENTER SINGLE USER MODE command description in 4.1/5.0/5.1 Manuals.