Bug #27766 DN restarts using ndb_mgm give inconsistent results
Submitted: 11 Apr 2007 19:55 Modified: 11 Apr 2007 19:58
Reporter: Jonathan Miller Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:mysql-5.1-telco-6.2 OS:Linux (32 Bit)
Assigned to: CPU Architecture:Any
Tags: mysql-5.1.18 ndb-6.2.1
Triage: Triaged: D3 (Medium)

[11 Apr 2007 19:55] Jonathan Miller
Description:
Hi,

In cleaning up today to restart stress testing, I noticed that I get inconsistent results when using mgm to restart a DN.

1) One restart never restarted
2) Many times the mgm never reported the data node had completed the restart
3) "Start initiated" sometimes displays and sometimes it does not, or does so out of order.

ndb_mgm> 2 restart -i
Shutting down nodes with "-n, no start" option, to subsequently start the nodes.
Node 2: Node shutdown initiated
Node 2: Node shutdown completed, restarting, no start, initial.
Node 2 is being restarted

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: started (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> 3 restart -i
Shutting down nodes with "-n, no start" option, to subsequently start the nodes.
Node 3: Node shutdown initiated
Node 3: Node shutdown completed, restarting, no start, initial.
Node 3 is being restarted

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: not started (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: not started (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> 3 start
Database node 3 is being started.

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: started (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> 3 restart -i
Shutting down nodes with "-n, no start" option, to subsequently start the nodes.
Node 3: Node shutdown completed, restarting, no start, initial.
Node 3 is being restarted

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: starting (Phase 3) (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> 3 restart
Shutting down nodes with "-n, no start" option, to subsequently start the nodes.
Node 3: Node shutdown completed, restarting, no start.
Node 3 is being restarted

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: started (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> 2 restart
Shutting down nodes with "-n, no start" option, to subsequently start the nodes.
Node 2: Node shutdown completed, restarting, no start.
Node 2 is being restarted

ndb_mgm> Node 2: Started (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> 3 restart
Shutting down nodes with "-n, no start" option, to subsequently start the nodes.
Node 3: Node shutdown initiated
Node 3: Node shutdown completed, restarting, no start.
Node 3 is being restarted

ndb_mgm> Node 3: Started (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> 3 restart
Shutting down nodes with "-n, no start" option, to subsequently start the nodes.
Node 3: Node shutdown initiated
Node 3: Node shutdown completed, restarting, no start.
Node 3 is being restarted

ndb_mgm> Node 3: Start initiated (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: starting (Phase 4) (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: starting (Phase 100) (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> Node 3: Started (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> 3 restart -i
Shutting down nodes with "-n, no start" option, to subsequently start the nodes.
Node 3: Node shutdown initiated
Node 3: Node shutdown completed, restarting, no start, initial.
Node 3 is being restarted

ndb_mgm> Node 3: Start initiated (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: starting (Phase 3) (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: starting (Phase 3) (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> all status
Node 2: started (mysql-5.1.18 ndb-6.2.1)
Node 3: starting (Phase 3) (mysql-5.1.18 ndb-6.2.1)

ndb_mgm> Node 3: Started (mysql-5.1.18 ndb-6.2.1)

How to repeat:
See above

Suggested fix:
Be consistent