-------------------------------------------------------------------------------------- ndb_1_cluster.log note: the problem develops once node 4 is killed -------------------------------------------------------------------------------------- 2005-10-31 16:02:41 [MgmSrvr] INFO -- Node 8: Prepare arbitrator node 1 [ticket=112d00014921f11d ] 2005-10-31 16:02:41 [MgmSrvr] INFO -- Node 4: Prepare arbitrator node 1 [ticket=112d00014921f11d ] 2005-10-31 16:02:41 [MgmSrvr] INFO -- Node 6: Prepare arbitrator node 1 [ticket=112d00014921f11d ] 2005-10-31 16:02:41 [MgmSrvr] INFO -- Node 3: Prepare arbitrator node 1 [ticket=112d00014921f11d ] 2005-10-31 16:02:41 [MgmSrvr] INFO -- Node 5: Prepare arbitrator node 1 [ticket=112d00014921f11d ] 2005-10-31 16:02:41 [MgmSrvr] INFO -- Node 7: Prepare arbitrator node 1 [ticket=112d00014921f11d ] 2005-10-31 16:02:41 [MgmSrvr] INFO -- Node 9: Prepare arbitrator node 1 [ticket=112d00014921f11d ] 2005-10-31 16:02:41 [MgmSrvr] INFO -- Node 2: Started arbitrator node 1 [ticket=112d00014921f11d ] 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 1: Node 4 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 6: Node 4 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 2: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 8: Node 4 Disconnected 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 8: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 6: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 5: Node 4 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 7: Node 4 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 9: Node 4 Disconnected 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 5: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 7: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 9: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 000000 0000000fee. 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 1: Node 2 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 6: Node 2 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 8: Node 2 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 3: Node 2 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 7: Node 2 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 5: Node 2 Disconnected 2005-10-31 16:03:42 [MgmSrvr] ALERT -- Node 9: Node 2 Disconnected 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 8: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 2 sig->failNo = 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 8: Communication to Node 2 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 8: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 6: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 2 sig->failNo = 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 6: Communication to Node 2 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 6: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 7: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 2 sig->failNo = 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 7: Communication to Node 2 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 7: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 3: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 2 sig->failNo = 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 3: Communication to Node 2 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 5: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 2 sig->failNo = 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 5: Communication to Node 2 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 5: Communication to Node 4 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 9: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 2 sig->failNo = 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 9: Communication to Node 2 closed 2005-10-31 16:03:42 [MgmSrvr] INFO -- Node 9: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 1: Node 8 Disconnected 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 6: Node 8 Disconnected 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 3: Node 8 Disconnected 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 5: Node 8 Disconnected 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 7: Node 8 Disconnected 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 9: Node 8 Disconnected 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 3: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 3 sig->failNo = 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 3: Communication to Node 2 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 3: Communication to Node 8 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 5: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 3 sig->failNo = 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 5: Communication to Node 2 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 5: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 5: Communication to Node 8 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 6: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 3 sig->failNo = 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 6: Communication to Node 2 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 6: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 6: Communication to Node 8 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 7: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 3 sig->failNo = 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 7: Communication to Node 2 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 7: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 7: Communication to Node 8 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 9: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 3 sig->failNo = 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 9: Communication to Node 2 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 9: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 9: Communication to Node 8 closed 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 1: Node 3 Disconnected 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 5: Node 3 Disconnected 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 9: Node 3 Disconnected 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 6: Node 3 Disconnected 2005-10-31 16:03:43 [MgmSrvr] ALERT -- Node 7: Node 3 Disconnected 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 6: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 4 sig->failNo = 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 6: Communication to Node 2 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 6: Communication to Node 3 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 6: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 6: Communication to Node 8 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 5: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 4 sig->failNo = 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 5: Communication to Node 2 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 5: Communication to Node 3 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 5: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 5: Communication to Node 8 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 7: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 4 sig->failNo = 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 7: Communication to Node 2 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 7: Communication to Node 3 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 7: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 7: Communication to Node 8 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 9: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 4 sig->failNo = 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 9: Communication to Node 2 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 9: Communication to Node 3 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 9: Communication to Node 4 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Node 9: Communication to Node 8 closed 2005-10-31 16:03:43 [MgmSrvr] INFO -- Mgmt server state: nodeid 8 freed, m_reserved_nodes 000000 0000000eee. 2005-10-31 16:03:43 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 000000 0000000ee6. 2005-10-31 16:03:43 [MgmSrvr] INFO -- Mgmt server state: nodeid 2 freed, m_reserved_nodes 000000 0000000ee2. 2005-10-31 16:03:44 [MgmSrvr] ALERT -- Node 1: Node 6 Disconnected 2005-10-31 16:03:44 [MgmSrvr] ALERT -- Node 7: Node 6 Disconnected 2005-10-31 16:03:44 [MgmSrvr] ALERT -- Node 5: Node 6 Disconnected 2005-10-31 16:03:44 [MgmSrvr] ALERT -- Node 9: Node 6 Disconnected 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 9: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 5 sig->failNo = 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 9: Communication to Node 2 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 9: Communication to Node 3 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 9: Communication to Node 4 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 9: Communication to Node 6 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 9: Communication to Node 8 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 5 sig->failNo = 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 2 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 3 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 4 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 6 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 8 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 5 sig->failNo = 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 2 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 3 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 4 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 6 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 8 closed 2005-10-31 16:03:44 [MgmSrvr] ALERT -- Node 1: Node 9 Disconnected 2005-10-31 16:03:44 [MgmSrvr] ALERT -- Node 5: Node 9 Disconnected 2005-10-31 16:03:44 [MgmSrvr] ALERT -- Node 7: Node 9 Disconnected 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 6 sig->failNo = 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 2 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 3 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 4 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 6 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 8 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 7: Communication to Node 9 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_bloc kCommit = 1 c_blockCommitNo = 6 sig->failNo = 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 2 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 3 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 4 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 6 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 8 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Node 5: Communication to Node 9 closed 2005-10-31 16:03:44 [MgmSrvr] INFO -- Mgmt server state: nodeid 9 freed, m_reserved_nodes 000000 0000000ce2. 2005-10-31 16:03:44 [MgmSrvr] INFO -- Mgmt server state: nodeid 6 freed, m_reserved_nodes 000000 0000000ca2. 2005-10-31 16:03:45 [MgmSrvr] ALERT -- Node 1: Node 7 Disconnected 2005-10-31 16:03:45 [MgmSrvr] ALERT -- Node 1: Node 5 Disconnected 2005-10-31 16:03:45 [MgmSrvr] INFO -- Mgmt server state: nodeid 7 freed, m_reserved_nodes 000000 0000000c22. 2005-10-31 16:03:45 [MgmSrvr] INFO -- Mgmt server state: nodeid 5 freed, m_reserved_nodes 000000 0000000c02.