Bug #31056 ndb_mgm report err after restart NDB node after enter single user mode
Submitted: 17 Sep 2007 12:18 Modified: 14 May 2009 13:00
Reporter: justin he Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:5.1 OS:Any
Assigned to: Magnus Blåudd CPU Architecture:Any

[17 Sep 2007 12:18] justin he
Description:
[justin.he@dev3-240 src]$ mgmclient/ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: 172.16.70.240:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2, Nodegroup: 0, Master)
id=3    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2)

[mysqld(API)]   6 node(s)
id=4    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2)
id=5    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)

ndb_mgm> enter single user mode 4;
Single user mode entered
Access is granted for API node 4 only.
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2, single user mode, Nodegroup: 0, Master)
id=3    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2, single user mode, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2)

[mysqld(API)]   6 node(s)
id=4    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2)
id=5    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)

ndb_mgm> 3 restart
Shutting down nodes with "-n, no start" option, to subsequently start the nodes.
Node 3: Node shutdown initiated
Node 3: Node shutdown completed, restarting, no start.
Node 3 is being restarted

ndb_mgm> Node 3: Forced node shutdown completed. Occured during startphase 1. Initiated by
signal 6. Caused by error 2314:'Data node is not allowed to get added to the cluster while
it is in single user mode(Restart error). Temporary error, restart node'.

ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2, single user mode, Nodegroup: 0, Master)
id=3 (not connected, accepting connect from 172.16.70.240)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @172.16.70.240  (mysql-5.1.20 ndb-6.3.2)

[mysqld(API)]   6 node(s)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)
id=6 (not connected, accepting connect from any host)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)

How to repeat:
1. enter user mode
2. restart a NDB node, report error
[19 Sep 2007 7:43] Hartmut Holzgraefe
Verified, it's expected behavior though, but i'm not sure whether it is documented. 

So this is either a documentation issue, or we could make it a feature request to have ndb_mgm refuse stop or restart commands while in single user mode up front ...
[30 Apr 2008 3:56] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/46219

ChangeSet@1.2578, 2008-04-30 11:27:58+00:00, lzhou@dev3-63.(none) +1 -0
  BUG#31056 Limited node restart in sigle user mode
[10 Dec 2008 18:08] Magnus Blåudd
Patch looks fine and gives a clear error message. I think we should also check the same thing inside the management server and refuse the restart in the same way as we check if any nod is already starting. What do you say?

See:
storage/ndb/src/mgmsrv/MgmtSrvr.cpp>>
int MgmtSrvr::restartNodes(const Vector<NodeId> &node_ids,
                           int * stopCount, bool nostart,
                           bool initialStart, bool abort,
                           int *stopSelf)
{
  /*
    verify that no nodes are starting before stopping as this would cause
    the starting node to shutdown
  */
  if (!abort && check_nodes_starting())
    return OPERATION_NOT_ALLOWED_START_STOP;
   ^ Magnus, add "check_nodes_single_user" here.
[10 Dec 2008 18:10] Magnus Blåudd
+  for (int i = 0; i<cl->no_of_nodes; i++)
+    if((cl->node_states+i)->node_status == NDB_MGM_NODE_STATUS_SINGLEUSER)
+    {
+      ndbout_c("Cannot restart nodes: single user mode");
+      printError();

magnus: no need to call "printError" here
[10 Dec 2008 18:11] Magnus Blåudd
Also add testcase to storage/ndb/test/ndbapi/testSingleUserMode.cpp
[12 Dec 2008 10:52] Magnus Blåudd
we split testMgm.cpp into testMgm.cpp and testSingleUserMode.cpp in 6.4. In 6.3 you will have to add the testcase to testMgm.cpp and then move it when merging to 6.4
[12 Dec 2008 17:51] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61536

2797 Leonard Zhou	2008-12-12
      BUG#31056 Limited node restart in sigle user mode
[16 Dec 2008 10:37] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61744

2797 Leonard Zhou	2008-12-16
      BUG#31056 Limited node restart in sigle user mode
[17 Dec 2008 6:42] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61836

2803 Leonard Zhou	2008-12-17
      BUG#31056 Limited node restart in single user mode
[17 Dec 2008 6:46] li zhou
Approved by Magnus
[17 Dec 2008 7:57] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61837

2803 Leonard Zhou	2008-12-17
      BUG#31056 Limited node restart in single user mode
[17 Dec 2008 8:22] Bugs System
Pushed into 5.1.30-ndb-6.4.0 (revid:zhou.li@sun.com-20081217080647-c1q876zqinikfgxi) (version source revid:zhou.li@sun.com-20081217080647-c1q876zqinikfgxi) (merge vers: 5.1.30-ndb-6.4.0) (pib:6)
[17 Dec 2008 8:29] Bugs System
Pushed into 5.1.30-ndb-6.3.21 (revid:zhou.li@sun.com-20081217075205-uzzrfkeshp82oobe) (version source revid:zhou.li@sun.com-20081217075205-uzzrfkeshp82oobe) (merge vers: 5.1.30-ndb-6.3.21) (pib:6)
[2 Apr 2009 8:27] Jon Stephens
Hi Li,

A couple of questions:

It appears that, following the restart, the cluster is no longer in Single User Mode.

ndb_mgm> enter single user mode 20
Single user mode entered
Access is granted for API node 20 only.
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @127.0.0.1  (mysql-5.1.32 ndb-7.0.5, single user mode, Nodegroup: 0, Master)
id=2    @127.0.0.1  (mysql-5.1.32 ndb-7.0.5, single user mode, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=10   @127.0.0.1  (mysql-5.1.32 ndb-7.0.5)

[mysqld(API)]   2 node(s)
id=20   @127.0.0.1  (mysql-5.1.32 ndb-7.0.5)
id=21 (not connected, accepting connect from any host)

ndb_mgm> all status
Node 1: single user mode (mysql-5.1.32 ndb-7.0.5)
Node 2: single user mode (mysql-5.1.32 ndb-7.0.5)

ndb_mgm> all restart
Executing RESTART on all nodes.
Starting shutdown. This may take a while. Please wait...
Node 1: Cluster shutdown initiated
Node 2: Cluster shutdown initiated
Node 2: Node shutdown completed, restarting, no start.
Node 1: Node shutdown completed, restarting, no start.
All DB nodes are being restarted.

ndb_mgm> Node 2: Start initiated (version 7.0.5)
Node 2: Data usage decreased to 0%(0 32K pages of total 3200)
Node 1: Start initiated (version 7.0.5)
Node 1: Data usage decreased to 0%(0 32K pages of total 3200)

ndb_mgm> all status
Node 1: starting (Last completed phase 3) (mysql-5.1.32 ndb-7.0.5)
Node 2: starting (Last completed phase 3) (mysql-5.1.32 ndb-7.0.5)

Node 1: Started (version 7.0.5)
Node 2: Started (version 7.0.5)

ndb_mgm> all status
Node 1: started (mysql-5.1.32 ndb-7.0.5)
Node 2: started (mysql-5.1.32 ndb-7.0.5)

ndb_mgm> exit single user mode
Exiting single user mode in progress.
Use ALL STATUS or SHOW to see when single user mode has been exited.
ndb_mgm> all status
Node 1: started (mysql-5.1.32 ndb-7.0.5)
Node 2: started (mysql-5.1.32 ndb-7.0.5)

I also see that I can EXIT SINGLE USER MODE even though it doesn't look like the cluster was actually in single user mode to start with.

Is this also intended?
[2 Apr 2009 8:32] Jon Stephens
Added an entry to the NDB-6.3.21 and 6.4.0 changelogs:

        It is now possible while in Single User Mode to restart all data
        nodes using ALL RESTART in the management client. Restarting of
        individual nodes while in Single User Mode remains disallowed.

However, I need the answered the questions I raised in my previous comment before I can update the sections of the Manual covering RESTART and single user mode, and close this bug.
[14 May 2009 13:00] Jon Stephens
Added note to Cluster Management Client Commands section (http://lists.mysql.com/commits/74046).

Closed.