Bug #21258 Cluster will not start if new nodes added to the cluster with existing database.
Submitted: 24 Jul 2006 19:38 Modified: 27 Jul 2006 12:31
Reporter: Nikolay Grishakin Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:5.1 OS:Linux (Linux)
Assigned to: CPU Architecture:Any
Tags: Test blocking

[24 Jul 2006 19:38] Nikolay Grishakin 
Description:
Tested adding new data nodes to the existing cluster. I used 1 node group and expand it to 2 node group cluster. I populated database in one node cluster and then shutdown whole cluster and started 2 node configuration. I did not use –initial parameter because I have existing database. 2 node cluster failed to start. Errors I’ve got from ndbd: 
“Status: Permanent error, external action needed
Message: Invalid configuration received from Management Server (Configuration error)
Error: 2350 Error data: Illegal configuration change. Initial start needs to be performed  when changing no of storage nodes (node 4) Error object: DBDIH (Line: 1559) 0x0000000a
Program: /home/ndbdev/ngrishakin/builds/libexec/ndbd Pid: 1710 Trace: /space/run/ndb_2_trace.log.1
Version: Version 5.1.12 (beta)
***EOM***” 

--------------- Signal ----------------
r.bn: 246 "DBDIH", r.proc: 2, r.sigId: 1809426 gsn: 348 "READ_NODESCONF" prio: 1
s.bn: 251 "NDBCNTR", s.proc: 2, s.sigId: 1809425 length: 13 trace: 0 #sec: 0 fragInf: 0
 noOfNodes: 4
 ndynamicId: 1
 masterNodeId: 2
 allNodes(defined): 000000000000003c
 inactiveNodes: 0000000000000030
 clusterNodes: 000000000000003c
 startedNodes: 0000000000000000
 startingNodes: 000000000000000c
--------------- Signal ----------------
r.bn: 251 "NDBCNTR", r.proc: 2, r.sigId: 1809425 gsn: 350 "READ_NODESREQ" prio: 1
s.bn: 246 "DBDIH", s.proc: 2, s.sigId: 1809424 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00f60002
--------------- Signal ----------------
r.bn: 246 "DBDIH", r.proc: 2, r.sigId: 1809424 gsn: 188 "UTIL_CREATE_LOCK_CONF" prio: 1
s.bn: 256 "DBUTIL", s.proc: 2, s.sigId: 1809423 length: 3 trace: 0 #sec: 0 fragInf: 0
 senderData: 0
 senderRef: 1000002
 lockId: 1
--------------- Signal ----------------
r.bn: 256 "DBUTIL", r.proc: 2, r.sigId: 1809423 gsn: 132 "UTIL_CREATE_LOCK_REQ" prio: 1
s.bn: 246 "DBDIH", s.proc: 2, s.sigId: 1809422 length: 4 trace: 0 #sec: 0 fragInf: 0
 senderData: 0
 senderRef: f60002
 lockId: 1
 lockType: 0
--------------- Signal ----------------
r.bn: 246 "DBDIH", r.proc: 2, r.sigId: 1809422 gsn: 188 "UTIL_CREATE_LOCK_CONF" prio: 1
s.bn: 256 "DBUTIL", s.proc: 2, s.sigId: 1809421 length: 3 trace: 0 #sec: 0 fragInf: 0
 senderData: 0
 senderRef: 1000002
 lockId: 0
--------------- Signal ----------------
r.bn: 256 "DBUTIL", r.proc: 2, r.sigId: 1809421 gsn: 132 "UTIL_CREATE_LOCK_REQ" prio: 1
s.bn: 246 "DBDIH", s.proc: 2, s.sigId: 1809420 length: 4 trace: 0 #sec: 0 fragInf: 0
 senderData: 0
 senderRef: f60002
 lockId: 0
 lockType: 0

How to repeat:
see privat comments.
[25 Jul 2006 0:18] Nikolay Grishakin 
The same result using "perl load_tpcb.pl -so -e ndb" instead of ndbmix.

Time: Tuesday 25 July 2006 - 02:14:30
Status: Permanent error, external action needed
Message: Invalid configuration received from Management Server (Configuration error)
Error: 2350
Error data: Illegal configuration change. Initial start needs to be performed  when changing no of s
torage nodes (node 4)
Error object: DBDIH (Line: 1559) 0x0000000a
Program: /home/ndbdev/ngrishakin/builds/libexec/ndbd
Pid: 16206
Trace: /space/run/ndb_3_trace.log.1
Version: Version 5.1.12 (beta)
***EOM***
[27 Jul 2006 12:32] Hartmut Holzgraefe 
When changing the node configuration (number of nodes, number of replicas) you need to backup the cluster, shutdown and restart with --initial and restore, 
else the data distribution doesn't match the new hash distribution.

This will only change with the "Online add nodes" feature