2006-11-07 00:29:17 [MgmSrvr] INFO     -- Node 4: Local checkpoint 422 started. Keep GCI = 14972 oldest restorable GCI = 14991
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 1: Node 6 Connected
2006-11-07 00:29:27 [MgmSrvr] ALERT    -- Node 4: Node 6 Disconnected
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 4: Communication to Node 6 closed
2006-11-07 00:29:27 [MgmSrvr] ALERT    -- Node 5: Node 6 Disconnected
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 5: Communication to Node 6 closed
2006-11-07 00:29:27 [MgmSrvr] ALERT    -- Node 7: Node 6 Disconnected
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 7: Communication to Node 6 closed
2006-11-07 00:29:27 [MgmSrvr] ALERT    -- Node 4: Arbitration check won - node group majority
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 4: President restarts arbitration thread [state=6]
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 4: DICT: lock bs: 0 ops: 0 poll: 0 cnt: 0 queue:
2006-11-07 00:29:27 [MgmSrvr] ALERT    -- Node 4: Node 3 Disconnected
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 4: Communication to Node 3 closed
2006-11-07 00:29:27 [MgmSrvr] ALERT    -- Node 5: Node 3 Disconnected
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 5: Communication to Node 3 closed
2006-11-07 00:29:27 [MgmSrvr] ALERT    -- Node 7: Node 3 Disconnected
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 7: Communication to Node 3 closed
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Mgmt server state: nodeid 8 reserved for ip 10.0.0.172, m_reserved_nodes 0000000000000102.
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 8: mysqld --server-id=0
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 4: Node 8 Connected
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 7: Node 8 Connected
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 5: Node 8 Connected
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 5: Node 8: API version 5.0.24
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 7: Node 8: API version 5.0.24
2006-11-07 00:29:27 [MgmSrvr] INFO     -- Node 4: Node 8: API version 5.0.24
2006-11-07 00:29:30 [MgmSrvr] INFO     -- Node 7: Communication to Node 3 opened
2006-11-07 00:29:30 [MgmSrvr] INFO     -- Node 4: Communication to Node 3 opened
2006-11-07 00:29:31 [MgmSrvr] INFO     -- Node 5: Communication to Node 3 opened
2006-11-07 00:29:33 [MgmSrvr] INFO     -- Node 7: Communication to Node 6 opened
2006-11-07 00:29:33 [MgmSrvr] INFO     -- Node 4: Communication to Node 6 opened
2006-11-07 00:29:34 [MgmSrvr] INFO     -- Node 5: Communication to Node 6 opened
2006-11-07 00:30:47 [MgmSrvr] INFO     -- Node 4: Local checkpoint 423 started. Keep GCI = 15031 oldest restorable GCI = 15049
2006-11-07 00:30:50 [MgmSrvr] INFO     -- Node 1: Node 5 Connected
2006-11-07 00:30:50 [MgmSrvr] ALERT    -- Node 7: Node 5 Disconnected
2006-11-07 00:30:50 [MgmSrvr] ALERT    -- Node 4: Node 5 Disconnected
2006-11-07 00:30:50 [MgmSrvr] INFO     -- Node 4: Communication to Node 5 closed
2006-11-07 00:30:50 [MgmSrvr] INFO     -- Node 4: Communication to Node 5 closed
2006-11-07 00:30:50 [MgmSrvr] ALERT    -- Node 4: Network partitioning - arbitration required
2006-11-07 00:30:50 [MgmSrvr] INFO     -- Node 4: President restarts arbitration thread [state=7]
2006-11-07 00:30:50 [MgmSrvr] INFO     -- Node 7: Communication to Node 5 closed
2006-11-07 00:30:50 [MgmSrvr] ALERT    -- Node 4: Arbitration won - positive reply from node 1
2006-11-07 00:30:50 [MgmSrvr] INFO     -- Node 4: DICT: lock bs: 0 ops: 0 poll: 0 cnt: 0 queue:
2006-11-07 00:30:50 [MgmSrvr] INFO     -- Node 7: Prepare arbitrator node 1 [ticket=7aa50002bfd310e9]
2006-11-07 00:30:51 [MgmSrvr] INFO     -- Node 4: Started arbitrator node 1 [ticket=7aa50002bfd310e9]
2006-11-07 00:30:56 [MgmSrvr] INFO     -- Node 7: Communication to Node 5 opened
2006-11-07 00:30:56 [MgmSrvr] INFO     -- Node 4: Communication to Node 5 opened
2006-11-07 00:31:21 [MgmSrvr] INFO     -- Node 1: Node 4 Connected
2006-11-07 00:31:22 [MgmSrvr] INFO     -- Node 1: Node 7 Connected
2006-11-07 00:31:22 [MgmSrvr] INFO     -- Mgmt server state: nodeid 8 freed, m_reserved_nodes 0000000000000002.
2006-11-07 00:31:22 [MgmSrvr] INFO     -- Mgmt server state: nodeid 3 reserved for ip 10.0.0.172, m_reserved_nodes 000000000000000a.
2006-11-07 00:31:22 [MgmSrvr] INFO     -- Node 3: mysqld --server-id=0
2006-11-07 00:31:22 [MgmSrvr] INFO     -- Mgmt server state: nodeid 2 reserved for ip 10.0.0.170, m_reserved_nodes 000000000000000e.
2006-11-07 00:31:22 [MgmSrvr] INFO     -- Node 2: mysqld --server-id=0

at this point whole cluster was down so I restarted mgmd and tried to start one ndbd from each node group.

2006-11-07 00:34:26 [MgmSrvr] INFO     -- Restarting server...
2006-11-07 00:34:28 [MgmSrvr] INFO     -- Mgmt server state: nodeid 2 freed, m_reserved_nodes 000000000000000a.
2006-11-07 00:34:28 [MgmSrvr] INFO     -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 0000000000000002.
2006-11-07 00:34:28 [MgmSrvr] INFO     -- Shutdown complete
2006-11-07 00:34:29 [MgmSrvr] INFO     -- Mgmt server state: nodeid 1 reserved for ip 10.0.0.170, m_reserved_nodes 0000000000000002.
2006-11-07 00:34:29 [MgmSrvr] INFO     -- Node 1: Node 1 Connected
2006-11-07 00:34:29 [MgmSrvr] INFO     -- NDB Cluster Management Server. Version 5.0.24
2006-11-07 00:34:29 [MgmSrvr] INFO     -- NDB Cluster Management Server. Version 5.0.24
2006-11-07 00:34:29 [MgmSrvr] INFO     -- Id: 1, Command port: 1186
2006-11-07 00:34:29 [MgmSrvr] INFO     -- Id: 1, Command port: 1186
2006-11-07 00:35:34 [MgmSrvr] INFO     -- Mgmt server state: nodeid 4 reserved for ip 10.0.0.170, m_reserved_nodes 0000000000000012.
2006-11-07 00:35:34 [MgmSrvr] INFO     -- Mgmt server state: nodeid 4 reserved for ip 10.0.0.170, m_reserved_nodes 0000000000000012.
2006-11-07 00:35:35 [MgmSrvr] INFO     -- Node 1: Node 4 Connected
2006-11-07 00:35:35 [MgmSrvr] INFO     -- Node 1: Node 4 Connected
2006-11-07 00:35:35 [MgmSrvr] INFO     -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000002.
2006-11-07 00:35:35 [MgmSrvr] INFO     -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000002.
2006-11-07 00:35:38 [MgmSrvr] INFO     -- Node 4: Waiting 27 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:38 [MgmSrvr] INFO     -- Node 4: Waiting 27 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:41 [MgmSrvr] INFO     -- Node 4: Waiting 24 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:41 [MgmSrvr] INFO     -- Node 4: Waiting 24 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:44 [MgmSrvr] INFO     -- Node 4: Waiting 21 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:44 [MgmSrvr] INFO     -- Node 4: Waiting 21 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:47 [MgmSrvr] INFO     -- Node 4: Waiting 18 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:47 [MgmSrvr] INFO     -- Node 4: Waiting 18 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:50 [MgmSrvr] INFO     -- Node 4: Waiting 15 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:50 [MgmSrvr] INFO     -- Node 4: Waiting 15 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:53 [MgmSrvr] INFO     -- Node 4: Waiting 12 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:53 [MgmSrvr] INFO     -- Node 4: Waiting 12 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait
: 0000000000000000 ]
2006-11-07 00:35:56 [MgmSrvr] INFO     -- Node 4: Waiting 9 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait:
 0000000000000000 ]
2006-11-07 00:35:56 [MgmSrvr] INFO     -- Node 4: Waiting 9 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait:
 0000000000000000 ]
2006-11-07 00:35:59 [MgmSrvr] INFO     -- Node 4: Waiting 6 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait:
 0000000000000000 ]
2006-11-07 00:35:59 [MgmSrvr] INFO     -- Node 4: Waiting 6 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait:
 0000000000000000 ]
2006-11-07 00:36:02 [MgmSrvr] INFO     -- Node 4: Waiting 3 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait:
 0000000000000000 ]
2006-11-07 00:36:02 [MgmSrvr] INFO     -- Node 4: Waiting 3 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait:
 0000000000000000 ]
2006-11-07 00:36:05 [MgmSrvr] INFO     -- Node 4: Waiting 0 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait:
 0000000000000000 ]
2006-11-07 00:36:05 [MgmSrvr] INFO     -- Node 4: Waiting 0 sec for nodes 00000000000000e0 to connect, nodes [ all: 00000000000000f0 connected: 0000000000000010 no-wait:
 0000000000000000 ]
2006-11-07 00:36:08 [MgmSrvr] INFO     -- Node 1: Node 4 Connected
2006-11-07 00:36:08 [MgmSrvr] INFO     -- Node 1: Node 4 Connected
2006-11-07 00:36:09 [MgmSrvr] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 1. Initiated by signal 0. Caused by error 2311: 'Conflict whe
n selecting restart type(Internal error, programming error or missing error message, please report a bug). Temporary error, rest
2006-11-07 00:36:09 [MgmSrvr] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 1. Initiated by signal 0. Caused by error 2311: 'Conflict whe
n selecting restart type(Internal error, programming error or missing error message, please report a bug). Temporary error, rest

and so on.