[Node 3 – Nodegroup  - 0]
Time: Friday 13 April 2012 - 07:57:31
Status: Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 5971) 0x00000002
Program: ndbmtd
Pid: 24992 thr: 0
Version: mysql-5.1.56 ndb-7.1.18


2012-04-13 07:56:16 [ndbd] INFO     -- findNeighbours from: 4857 old (left: 6 right: 5) new (4 5)
2012-04-13 07:57:31 [ndbd] INFO     -- findNeighbours from: 4857 old (left: 4 right: 5) new (5 5)
2012-04-13 07:57:31 [ndbd] INFO     -- Arbitrator decided to shutdown this node
2012-04-13 07:57:31 [ndbd] INFO     -- QMGR (Line: 5971) 0x00000002
2012-04-13 07:57:31 [ndbd] INFO     -- Error handler shutting down system
2012-04-13 07:57:31 [ndbd] INFO     -- Error handler shutdown completed – exiting
2012-04-13 07:57:32 [ndbd] ALERT    -- Node 3: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'.













[Node 4 – Nodegroup - 1]
Time: Friday 13 April 2012 - 07:57:35
Status: Temporary error, restart node
Message: Node declared dead. See error log for details (Arbitration error)
Error: 2315
Error data: We(4) have been declared dead by 3 (via 5) reason: Heartbeat failure(4)
Error object: QMGR (Line: 3930) 0x00000002
Program: ndbmtd
Pid: 21856 thr: 0
Version: mysql-5.1.56 ndb-7.1.18
Trace: /rancore1/mysql-cluster/data0/ndb_4_trace.log.10 /rancore1/mysql-cluster/data0/ndb_4_trace.log.10_t1 /rancore1/mysql-cluster/data0/ndb


2012-04-13 07:56:19 [ndbd] INFO     -- findNeighbours from: 4857 old (left: 5 right: 6) new (5 3)
2012-04-13 07:57:35 [ndbd] INFO     -- We(4) have been declared dead by 3 (via 5) reason: Heartbeat failure(4)
2012-04-13 07:57:35 [ndbd] INFO     -- QMGR (Line: 3930) 0x00000002
2012-04-13 07:57:35 [ndbd] INFO     -- Error handler shutting down system
2012-04-13 07:57:35 [ndbd] INFO     -- Error handler shutdown completed – exiting
2012-04-13 07:57:36 [ndbd] ALERT    -- Node 4: Forced node shutdown completed. Caused by error 2315: 'Node declared dead. See error log for details(Arbitration error). Temporary error, restart node'.

















[Node 5 – Nodegroup - 0]
Time: Friday 13 April 2012 - 07:57:32
Status: Temporary error, restart node
Message: Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 5971) 0x00000002
Program: ndbmtd
Pid: 24999 thr: 0
Version: mysql-5.1.56 ndb-7.1.18
Trace: /rancore2/mysql-cluster/data2/ndb_5_trace.log.7 /rancore2/mysql-cluster/dat


2012-04-13 07:56:16 [ndbd] INFO     -- findNeighbours from: 4857 old (left: 3 right: 4) new (3 4)
2012-04-13 07:57:31 [ndbd] INFO     -- findNeighbours from: 4857 old (left: 3 right: 4) new (3 3)
2012-04-13 07:57:32 [ndbd] INFO     -- findNeighbours from: 4857 old (left: 3 right: 3) new (65535 65535)
2012-04-13 07:57:32 [ndbd] INFO     -- Arbitrator decided to shutdown this node
2012-04-13 07:57:32 [ndbd] INFO     -- QMGR (Line: 5971) 0x00000002
2012-04-13 07:57:32 [ndbd] INFO     -- Error handler shutting down system
2012-04-13 07:57:32 [ndbd] INFO     -- Error handler shutdown completed – exiting
2012-04-13 07:57:33 [ndbd] ALERT    -- Node 5: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'.














 
[Node 6 – Nodegroup – 1]
Time: Friday 13 April 2012 - 07:56:19
Status: Temporary error, restart node
Message: Node declared dead. See error log for details (Arbitration error)
Error: 2315
Error data: We(6) have been declared dead by 3 (via 5) reason: Heartbeat failure(4)
Error object: QMGR (Line: 3930) 0x00000002
Program: ndbmtd
Pid: 21862 thr: 0
Version: mysql-5.1.56 ndb-7.1.18
Trace: /rancore2/mysql-cluster/data2/ndb_6_trace.log.3 /rancore2/mysql-cluster/data2/ndb_6_trace.log.3_t1 /rancore2/mysql-cluster/data2/ndb_6



2012-04-13 07:56:20 [ndbd] INFO     -- We(6) have been declared dead by 3 (via 5) reason: Heartbeat failure(4)
2012-04-13 07:56:20 [ndbd] INFO     -- QMGR (Line: 3930) 0x00000002
2012-04-13 07:56:20 [ndbd] INFO     -- Error handler shutting down system
2012-04-13 07:56:20 [ndbd] INFO     -- Error handler shutdown completed – exiting
2012-04-13 07:56:21 [ndbd] ALERT    -- Node 6: Forced node shutdown completed. Caused by error 2315: 'Node declared dead. See error log for details(Arbitration error). Temporary error, restart node'.