Bug #55622 ndbmtd node start causes all cluster to fail
Submitted: 29 Jul 2010 9:48 Modified: 8 Sep 2016 6:45
Reporter: Alexey Asemov Email Updates:
Status: Unsupported Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:mysql-5.1-telco-7.1 OS:Linux (CentOS 5.5)
Assigned to: CPU Architecture:Any
Tags: mysql-5.1.44 ndb-7.1.4b

[29 Jul 2010 9:48] Alexey Asemov
Description:
I am using NDB cluster with 3 NDB nodes and 3 replicas (full "RAID1" over three servers). 

When I stop one of the nodes (or when it is failed due to power outage) I can't restart it because all the data nodes will fail on the node start.

The problem is: when cluster is completely restarted (shutdown and restart), all nodes start correctly. When one node is hot-started, all nodes fail, and cluster goes into full restart mode. This repeats few more times after that, and only then the cluster starts.

How to repeat:
Make three nodes with 3 replicas. Stop one of the ndbmtd data nodes. Start it. All other nodes will fail.
[29 Jul 2010 10:08] Alexey Asemov
bug-data-55622.zip uploaded to the bug data upload FTP, it contains all the tracebacks.
[10 Aug 2010 13:17] Hartmut Holzgraefe
Running with NoOfReplicas=3 or 4 is not well tested and is known to have bugs, we currently only actually support NoOfReplicas=1 (for pure speed, no HA) and =2 (for HA) ...
[16 Aug 2010 10:59] Hartmut Holzgraefe
Can you reproduce the same with NoOfReplicas=2 ?
[26 Aug 2010 7:01] Alexey Asemov
Transitioning to two-replica ndbmtd cluster got me with plenty of LCP failures with NDBFS error code 1217. 

Transitioning to ndbd solved LCP failures, and is now running smooth. I will try a bit of node restarts in the meanwhile and report accordingly.
[24 Sep 2010 7:29] Alexey Asemov
May be related to #54945? Can I get 7.1.8 sources for test?
[8 Sep 2016 6:45] MySQL Verification Team
> 2 replica's is still unsupported. It's experimental still with latest versions of the ndbcluster. What is reported here is still more/less expected behavior.

kind regards
Bogdan Kecman