2007-07-05 06:46:38 [MgmSrvr] INFO -- NDB Cluster Management Server. Version 5.1.19 (beta) 2007-07-05 06:46:38 [MgmSrvr] INFO -- Id: 1, Command port: 14000 2007-07-05 06:46:42 [MgmSrvr] INFO -- Mgmt server state: nodeid 2 reserved for ip 127.0.0.1, m_reserved_nodes 0000000000000006. 2007-07-05 06:46:42 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 reserved for ip 127.0.0.1, m_reserved_nodes 000000000000000e. 2007-07-05 06:46:43 [MgmSrvr] INFO -- Node 1: Node 2 Connected 2007-07-05 06:46:43 [MgmSrvr] INFO -- Mgmt server state: nodeid 2 freed, m_reserved_nodes 000000000000000a. 2007-07-05 06:46:43 [MgmSrvr] INFO -- Node 1: Node 3 Connected 2007-07-05 06:46:44 [MgmSrvr] INFO -- Node 2: Node 3 Connected 2007-07-05 06:46:44 [MgmSrvr] INFO -- Node 2: Initial start with nodes 000000000000000c [ missing: 0000000000000000 no-wait: 0000000000000000 ] 2007-07-05 06:46:44 [MgmSrvr] INFO -- Node 2: CM_REGCONF president = 2, own Node = 2, our dynamic id = 1 2007-07-05 06:46:44 [MgmSrvr] INFO -- Node 3: Node 2 Connected 2007-07-05 06:46:44 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 0000000000000002. 2007-07-05 06:46:47 [MgmSrvr] INFO -- Node 2: Node 3: API version 5.1.19 2007-07-05 06:46:47 [MgmSrvr] INFO -- Node 2: Start phase 1 completed 2007-07-05 06:46:47 [MgmSrvr] INFO -- Node 3: CM_REGCONF president = 2, own Node = 3, our dynamic id = 2 2007-07-05 06:46:47 [MgmSrvr] INFO -- Node 3: Node 2: API version 5.1.19 2007-07-05 06:46:47 [MgmSrvr] INFO -- Node 3: Start phase 1 completed 2007-07-05 06:46:47 [MgmSrvr] INFO -- Node 2: Start phase 2 completed (initial start) 2007-07-05 06:46:47 [MgmSrvr] INFO -- Node 3: Start phase 2 completed (initial start) 2007-07-05 06:46:47 [MgmSrvr] INFO -- Node 2: Start phase 3 completed (initial start) 2007-07-05 06:46:47 [MgmSrvr] INFO -- Node 3: Start phase 3 completed (initial start) 2007-07-05 06:48:01 [MgmSrvr] INFO -- Node 2: Start phase 4 completed (initial start) 2007-07-05 06:48:01 [MgmSrvr] INFO -- Node 3: Start phase 4 completed (initial start) 2007-07-05 06:48:03 [MgmSrvr] INFO -- Node 2: Local checkpoint 1 started. Keep GCI = 1 oldest restorable GCI = 1 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Start phase 5 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Start phase 6 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: President restarts arbitration thread [state=1] 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Start phase 5 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Start phase 6 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Start phase 7 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Start phase 7 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 4 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 5 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 6 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 7 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 8 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 9 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 10 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 11 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 12 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 13 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 14 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 15 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Communication to Node 16 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Start phase 8 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Start phase 9 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Start phase 100 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Start phase 101 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Started (version 5.1.19) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 5 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 6 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 7 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 8 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 9 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 10 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 11 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 12 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 13 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 14 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 15 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Communication to Node 16 opened 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Start phase 8 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Start phase 9 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Start phase 100 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Start phase 101 completed (initial start) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Started (version 5.1.19) 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Node 1: API version 5.1.19 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Node 1: API version 5.1.19 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 3: Prepare arbitrator node 1 [ticket=179a0001963559b3] 2007-07-05 06:48:05 [MgmSrvr] INFO -- Node 2: Started arbitrator node 1 [ticket=179a0001963559b3] 2007-07-05 06:48:13 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 reserved for ip 127.0.0.1, m_reserved_nodes 0000000000000012. 2007-07-05 06:48:13 [MgmSrvr] INFO -- Node 2: Node 4 Connected 2007-07-05 06:48:13 [MgmSrvr] INFO -- Node 2: Node 4: API version 5.1.19 2007-07-05 06:48:13 [MgmSrvr] INFO -- Node 3: Node 4 Connected 2007-07-05 06:48:13 [MgmSrvr] INFO -- Node 3: Node 4: API version 5.1.19 2007-07-05 06:48:18 [MgmSrvr] INFO -- Node 3: Node shutdown initiated 2007-07-05 06:48:28 [MgmSrvr] ALERT -- Node 2: Node 3 Disconnected 2007-07-05 06:48:28 [MgmSrvr] INFO -- Node 2: Communication to Node 3 closed 2007-07-05 06:48:28 [MgmSrvr] ALERT -- Node 2: Network partitioning - arbitration required 2007-07-05 06:48:28 [MgmSrvr] INFO -- Node 2: President restarts arbitration thread [state=7] 2007-07-05 06:48:28 [MgmSrvr] ALERT -- Node 1: Node 3 Disconnected 2007-07-05 06:48:28 [MgmSrvr] INFO -- Node 3: Node shutdown completed, restarting, no start, initial. 2007-07-05 06:48:28 [MgmSrvr] ALERT -- Node 2: Arbitration won - positive reply from node 1 2007-07-05 06:48:28 [MgmSrvr] INFO -- Node 2: DICT: lock bs: 0 ops: 0 poll: 0 cnt: 0 queue: 2007-07-05 06:48:29 [MgmSrvr] INFO -- Node 2: Started arbitrator node 1 [ticket=179a00029635b5bc] 2007-07-05 06:48:29 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 reserved for ip 127.0.0.1, m_reserved_nodes 000000000000001a. 2007-07-05 06:48:30 [MgmSrvr] INFO -- Node 1: Node 3 Connected 2007-07-05 06:48:31 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 0000000000000012. 2007-07-05 06:48:32 [MgmSrvr] INFO -- Node 2: Communication to Node 3 opened 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 3: Start initiated (version 5.1.19) 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 3: Start phase 0 completed 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 3: Communication to Node 2 opened 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 3: Initial start, waiting for 0000000000000004 to connect, nodes [ all: 000000000000000c connected: 0000000000000008 no-wait: 0000000000000000 ] 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 2: Node 3 Connected 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 2: Node 3: API version 5.1.19 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 3: Node 2 Connected 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 3: CM_REGCONF president = 2, own Node = 3, our dynamic id = 3 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 3: Node 2: API version 5.1.19 2007-07-05 06:48:33 [MgmSrvr] INFO -- Node 3: Start phase 1 completed 2007-07-05 06:48:34 [MgmSrvr] INFO -- Node 3: Start phase 2 completed (initial node restart) 2007-07-05 06:48:34 [MgmSrvr] INFO -- Node 2: DICT: locked by node 3 for NodeRestart 2007-07-05 06:48:34 [MgmSrvr] INFO -- Node 2: DICT: lock bs: 4 ops: 0 poll: 0 cnt: 0 queue: 3L 2007-07-05 06:48:34 [MgmSrvr] INFO -- Node 3: Receive arbitrator node 1 [ticket=179a00029635b5bc] 2007-07-05 06:48:34 [MgmSrvr] INFO -- Node 3: Start phase 3 completed (initial node restart) 2007-07-05 06:49:21 [MgmSrvr] INFO -- Node 3: Start phase 4 completed (initial node restart) 2007-07-05 06:49:22 [MgmSrvr] INFO -- Node 2: Node: 3 StartLog: [GCI Keep: 1 LastCompleted: 11 NewestRestorable: 36] 2007-07-05 06:49:22 [MgmSrvr] INFO -- Node 3: Applying undo to LCP: 0 2007-07-05 06:49:22 [MgmSrvr] INFO -- Node 3: Undo head - undofile01.dat page: 1 lsn: 0 2007-07-05 06:49:23 [MgmSrvr] INFO -- Node 3: Logfile group: 2 2007-07-05 06:49:23 [MgmSrvr] INFO -- Node 3: head: undofile01.dat page: 1 2007-07-05 06:49:23 [MgmSrvr] INFO -- Node 3: tail: undofile01.dat page: 1 2007-07-05 06:49:23 [MgmSrvr] INFO -- Node 3: Flushing page cache after undo completion 2007-07-05 06:49:23 [MgmSrvr] INFO -- Node 3: Flushing complete 2007-07-05 06:49:23 [MgmSrvr] INFO -- Node 2: Local checkpoint 2 started. Keep GCI = 1 oldest restorable GCI = 2 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Suma: asking node 2 to recreate subscriptions on me 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Suma: node 2 has completed restoring me 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Start phase 5 completed (initial node restart) 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Start phase 6 completed (initial node restart) 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Start phase 7 completed (initial node restart) 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 5 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 6 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 7 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 8 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 9 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 10 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 11 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 12 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 13 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 14 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 15 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Communication to Node 16 opened 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Start phase 8 completed (initial node restart) 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Start phase 9 completed (initial node restart) 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Start phase 100 completed (initial node restart) 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Suma: initiate handover with nodes 0000000000000004 GCI: 40 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Suma: handover from node 2 gci: 40 buckets: 00000002 (2) 2007-07-05 06:49:24 [MgmSrvr] INFO -- Node 3: Node 1: API version 5.1.19 2007-07-05 06:49:25 [MgmSrvr] INFO -- Node 3: Node 4 Connected 2007-07-05 06:49:25 [MgmSrvr] INFO -- Node 3: Node 4: API version 5.1.19 2007-07-05 06:49:31 [MgmSrvr] INFO -- Node 2: DICT: unlocked by node 3 for NodeRestart 2007-07-05 06:49:31 [MgmSrvr] INFO -- Node 2: DICT: lock bs: 0 ops: 0 poll: 0 cnt: 0 queue: 2007-07-05 06:49:31 [MgmSrvr] INFO -- Node 3: Start phase 101 completed (initial node restart) 2007-07-05 06:49:31 [MgmSrvr] INFO -- Node 3: Started (version 5.1.19) 2007-07-05 06:49:34 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 06:49:34 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 06:49:34 [MgmSrvr] INFO -- Node 2: Communication to Node 4 closed 2007-07-05 06:49:34 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected 2007-07-05 06:49:34 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed 2007-07-05 06:49:35 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000002. 2007-07-05 06:49:37 [MgmSrvr] INFO -- Node 2: Communication to Node 4 opened 2007-07-05 06:49:38 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened 2007-07-05 07:26:51 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 reserved for ip 127.0.0.1, m_reserved_nodes 0000000000000012. 2007-07-05 07:26:51 [MgmSrvr] INFO -- Node 2: Node 4 Connected 2007-07-05 07:26:51 [MgmSrvr] INFO -- Node 2: Node 4: API version 5.1.19 2007-07-05 07:26:51 [MgmSrvr] INFO -- Node 3: Node 4 Connected 2007-07-05 07:26:51 [MgmSrvr] INFO -- Node 3: Node 4: API version 5.1.19 2007-07-05 07:27:00 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 07:27:00 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 07:27:00 [MgmSrvr] INFO -- Node 2: Communication to Node 4 closed 2007-07-05 07:27:00 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected 2007-07-05 07:27:00 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed 2007-07-05 07:27:00 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000002. 2007-07-05 07:27:03 [MgmSrvr] INFO -- Node 2: Communication to Node 4 opened 2007-07-05 07:27:04 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened 2007-07-05 07:27:14 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 reserved for ip 127.0.0.1, m_reserved_nodes 0000000000000012. 2007-07-05 07:27:14 [MgmSrvr] INFO -- Node 2: Node 4 Connected 2007-07-05 07:27:14 [MgmSrvr] INFO -- Node 2: Node 4: API version 5.1.19 2007-07-05 07:27:14 [MgmSrvr] INFO -- Node 3: Node 4 Connected 2007-07-05 07:27:14 [MgmSrvr] INFO -- Node 3: Node 4: API version 5.1.19 2007-07-05 07:27:22 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 07:27:22 [MgmSrvr] INFO -- Node 2: Communication to Node 4 closed 2007-07-05 07:27:22 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected 2007-07-05 07:27:22 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected 2007-07-05 07:27:22 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed 2007-07-05 07:27:23 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000002. 2007-07-05 07:27:26 [MgmSrvr] INFO -- Node 2: Communication to Node 4 opened 2007-07-05 07:27:26 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened 2007-07-05 07:27:52 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 reserved for ip 127.0.0.1, m_reserved_nodes 0000000000000012. 2007-07-05 07:27:52 [MgmSrvr] INFO -- Node 2: Node 4 Connected 2007-07-05 07:27:52 [MgmSrvr] INFO -- Node 2: Node 4: API version 5.1.19 2007-07-05 07:27:52 [MgmSrvr] INFO -- Node 3: Node 4 Connected 2007-07-05 07:27:52 [MgmSrvr] INFO -- Node 3: Node 4: API version 5.1.19 2007-07-05 07:27:58 [MgmSrvr] INFO -- Node 3: Node shutdown initiated 2007-07-05 07:28:08 [MgmSrvr] ALERT -- Node 2: Node 3 Disconnected 2007-07-05 07:28:08 [MgmSrvr] INFO -- Node 2: Communication to Node 3 closed 2007-07-05 07:28:08 [MgmSrvr] ALERT -- Node 2: Network partitioning - arbitration required 2007-07-05 07:28:08 [MgmSrvr] INFO -- Node 2: President restarts arbitration thread [state=7] 2007-07-05 07:28:08 [MgmSrvr] ALERT -- Node 1: Node 3 Disconnected 2007-07-05 07:28:08 [MgmSrvr] INFO -- Node 3: Node shutdown completed, restarting, no start, initial. 2007-07-05 07:28:08 [MgmSrvr] ALERT -- Node 2: Arbitration won - positive reply from node 1 2007-07-05 07:28:08 [MgmSrvr] INFO -- Node 2: DICT: lock bs: 0 ops: 0 poll: 0 cnt: 0 queue: 2007-07-05 07:28:09 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 reserved for ip 127.0.0.1, m_reserved_nodes 000000000000001a. 2007-07-05 07:28:09 [MgmSrvr] INFO -- Node 2: Started arbitrator node 1 [ticket=179a0003965a06d2] 2007-07-05 07:28:10 [MgmSrvr] INFO -- Node 1: Node 3 Connected 2007-07-05 07:28:10 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 0000000000000012. 2007-07-05 07:28:12 [MgmSrvr] INFO -- Node 2: Communication to Node 3 opened 2007-07-05 07:28:12 [MgmSrvr] INFO -- Node 3: Start initiated (version 5.1.19) 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: Start phase 0 completed 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: Communication to Node 2 opened 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: Initial start, waiting for 0000000000000004 to connect, nodes [ all: 000000000000000c connected: 0000000000000008 no-wait: 0000000000000000 ] 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 2: Node 3 Connected 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 2: Node 3: API version 5.1.19 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: Node 2 Connected 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: CM_REGCONF president = 2, own Node = 3, our dynamic id = 4 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: Node 2: API version 5.1.19 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: Start phase 1 completed 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: Receive arbitrator node 1 [ticket=179a0003965a06d2] 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: Start phase 2 completed (initial node restart) 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 2: DICT: locked by node 3 for NodeRestart 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 2: DICT: lock bs: 4 ops: 0 poll: 0 cnt: 0 queue: 3L 2007-07-05 07:28:13 [MgmSrvr] INFO -- Node 3: Start phase 3 completed (initial node restart) 2007-07-05 07:28:58 [MgmSrvr] INFO -- Node 3: Start phase 4 completed (initial node restart) 2007-07-05 07:29:00 [MgmSrvr] INFO -- Node 2: Node: 3 StartLog: [GCI Keep: 1 LastCompleted: 1070 NewestRestorable: 1094] 2007-07-05 07:29:00 [MgmSrvr] INFO -- Node 3: Applying undo to LCP: 0 2007-07-05 07:29:00 [MgmSrvr] INFO -- Node 3: Undo head - undofile01.dat page: 1 lsn: 0 2007-07-05 07:29:00 [MgmSrvr] INFO -- Node 3: Logfile group: 2 2007-07-05 07:29:00 [MgmSrvr] INFO -- Node 3: head: undofile01.dat page: 1 2007-07-05 07:29:00 [MgmSrvr] INFO -- Node 3: tail: undofile01.dat page: 1 2007-07-05 07:29:00 [MgmSrvr] INFO -- Node 3: Flushing page cache after undo completion 2007-07-05 07:29:00 [MgmSrvr] INFO -- Node 3: Flushing complete 2007-07-05 07:29:00 [MgmSrvr] INFO -- Node 2: Local checkpoint 3 started. Keep GCI = 36 oldest restorable GCI = 3 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Suma: asking node 2 to recreate subscriptions on me 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Suma: node 2 has completed restoring me 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Start phase 5 completed (initial node restart) 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Start phase 6 completed (initial node restart) 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Start phase 7 completed (initial node restart) 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 5 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 6 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 7 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 8 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 9 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 10 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 11 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 12 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 13 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 14 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 15 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Communication to Node 16 opened 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Start phase 8 completed (initial node restart) 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Start phase 9 completed (initial node restart) 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Start phase 100 completed (initial node restart) 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Suma: initiate handover with nodes 0000000000000004 GCI: 1098 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Suma: handover from node 2 gci: 1098 buckets: 00000002 (2) 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Node 1: API version 5.1.19 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Node 4 Connected 2007-07-05 07:29:02 [MgmSrvr] INFO -- Node 3: Node 4: API version 5.1.19 2007-07-05 07:29:09 [MgmSrvr] INFO -- Node 2: DICT: unlocked by node 3 for NodeRestart 2007-07-05 07:29:09 [MgmSrvr] INFO -- Node 2: DICT: lock bs: 0 ops: 0 poll: 0 cnt: 0 queue: 2007-07-05 07:29:09 [MgmSrvr] INFO -- Node 3: Start phase 101 completed (initial node restart) 2007-07-05 07:29:09 [MgmSrvr] INFO -- Node 3: Started (version 5.1.19) 2007-07-05 07:29:11 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 07:29:11 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 07:29:11 [MgmSrvr] INFO -- Node 2: Communication to Node 4 closed 2007-07-05 07:29:11 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected 2007-07-05 07:29:11 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed 2007-07-05 07:29:11 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000002. 2007-07-05 07:29:14 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened 2007-07-05 07:29:14 [MgmSrvr] INFO -- Node 2: Communication to Node 4 opened 2007-07-05 07:31:26 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 reserved for ip 127.0.0.1, m_reserved_nodes 0000000000000012. 2007-07-05 07:31:26 [MgmSrvr] INFO -- Node 2: Node 4 Connected 2007-07-05 07:31:26 [MgmSrvr] INFO -- Node 2: Node 4: API version 5.1.19 2007-07-05 07:31:26 [MgmSrvr] INFO -- Node 3: Node 4 Connected 2007-07-05 07:31:26 [MgmSrvr] INFO -- Node 3: Node 4: API version 5.1.19 2007-07-05 07:31:34 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 07:31:34 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 07:31:34 [MgmSrvr] INFO -- Node 2: Communication to Node 4 closed 2007-07-05 07:31:34 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected 2007-07-05 07:31:34 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed 2007-07-05 07:31:35 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000002. 2007-07-05 07:31:38 [MgmSrvr] INFO -- Node 2: Communication to Node 4 opened 2007-07-05 07:31:38 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened 2007-07-05 07:33:05 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 reserved for ip 127.0.0.1, m_reserved_nodes 0000000000000012. 2007-07-05 07:33:05 [MgmSrvr] INFO -- Node 3: Node 4 Connected 2007-07-05 07:33:05 [MgmSrvr] INFO -- Node 2: Node 4 Connected 2007-07-05 07:33:05 [MgmSrvr] INFO -- Node 2: Node 4: API version 5.1.19 2007-07-05 07:33:05 [MgmSrvr] INFO -- Node 3: Node 4: API version 5.1.19 2007-07-05 07:33:13 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 07:33:13 [MgmSrvr] INFO -- Node 2: Communication to Node 4 closed 2007-07-05 07:33:13 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected 2007-07-05 07:33:13 [MgmSrvr] ALERT -- Node 3: Node 4 Disconnected 2007-07-05 07:33:13 [MgmSrvr] INFO -- Node 3: Communication to Node 4 closed 2007-07-05 07:33:14 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000002. 2007-07-05 07:33:16 [MgmSrvr] INFO -- Node 2: Communication to Node 4 opened 2007-07-05 07:33:17 [MgmSrvr] INFO -- Node 3: Communication to Node 4 opened 2007-07-05 08:18:17 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 reserved for ip 127.0.0.1, m_reserved_nodes 0000000000000012. 2007-07-05 08:18:17 [MgmSrvr] INFO -- Node 2: Node 4 Connected 2007-07-05 08:18:17 [MgmSrvr] INFO -- Node 2: Node 4: API version 5.1.19 2007-07-05 08:18:17 [MgmSrvr] INFO -- Node 3: Node 4 Connected 2007-07-05 08:18:17 [MgmSrvr] INFO -- Node 3: Node 4: API version 5.1.19 2007-07-05 08:18:23 [MgmSrvr] INFO -- Node 3: Node shutdown initiated 2007-07-05 08:18:32 [MgmSrvr] ALERT -- Node 2: Node 3 Disconnected 2007-07-05 08:18:32 [MgmSrvr] INFO -- Node 2: Communication to Node 3 closed 2007-07-05 08:18:32 [MgmSrvr] ALERT -- Node 2: Network partitioning - arbitration required 2007-07-05 08:18:32 [MgmSrvr] INFO -- Node 2: President restarts arbitration thread [state=7] 2007-07-05 08:18:32 [MgmSrvr] ALERT -- Node 1: Node 3 Disconnected 2007-07-05 08:18:32 [MgmSrvr] INFO -- Node 3: Node shutdown completed, restarting, no start, initial. 2007-07-05 08:18:32 [MgmSrvr] ALERT -- Node 2: Arbitration won - positive reply from node 1 2007-07-05 08:18:32 [MgmSrvr] INFO -- Node 2: DICT: lock bs: 0 ops: 0 poll: 0 cnt: 0 queue: 2007-07-05 08:18:33 [MgmSrvr] INFO -- Node 2: Started arbitrator node 1 [ticket=179a000496882b7d] 2007-07-05 08:18:33 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 reserved for ip 127.0.0.1, m_reserved_nodes 000000000000001a. 2007-07-05 08:18:34 [MgmSrvr] INFO -- Node 1: Node 3 Connected 2007-07-05 08:18:35 [MgmSrvr] INFO -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 0000000000000012. 2007-07-05 08:18:36 [MgmSrvr] INFO -- Node 2: Communication to Node 3 opened 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 3: Start initiated (version 5.1.19) 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 3: Start phase 0 completed 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 3: Communication to Node 2 opened 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 3: Initial start, waiting for 0000000000000004 to connect, nodes [ all: 000000000000000c connected: 0000000000000008 no-wait: 0000000000000000 ] 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 2: Node 3 Connected 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 2: Node 3: API version 5.1.19 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 3: Node 2 Connected 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 3: CM_REGCONF president = 2, own Node = 3, our dynamic id = 5 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 3: Node 2: API version 5.1.19 2007-07-05 08:18:37 [MgmSrvr] INFO -- Node 3: Start phase 1 completed 2007-07-05 08:18:38 [MgmSrvr] INFO -- Node 3: Start phase 2 completed (initial node restart) 2007-07-05 08:18:38 [MgmSrvr] INFO -- Node 2: DICT: locked by node 3 for NodeRestart 2007-07-05 08:18:38 [MgmSrvr] INFO -- Node 2: DICT: lock bs: 4 ops: 0 poll: 0 cnt: 0 queue: 3L 2007-07-05 08:18:38 [MgmSrvr] INFO -- Node 3: Receive arbitrator node 1 [ticket=179a000496882b7d] 2007-07-05 08:18:38 [MgmSrvr] INFO -- Node 3: Start phase 3 completed (initial node restart) 2007-07-05 08:19:26 [MgmSrvr] INFO -- Node 3: Start phase 4 completed (initial node restart) 2007-07-05 08:19:27 [MgmSrvr] INFO -- Node 2: Node: 3 StartLog: [GCI Keep: 36 LastCompleted: 2414 NewestRestorable: 2439] 2007-07-05 08:19:27 [MgmSrvr] INFO -- Node 3: Applying undo to LCP: 0 2007-07-05 08:19:27 [MgmSrvr] ALERT -- Node 2: Node 3 Disconnected 2007-07-05 08:19:27 [MgmSrvr] INFO -- Node 2: Communication to Node 3 closed 2007-07-05 08:19:27 [MgmSrvr] ALERT -- Node 2: Network partitioning - arbitration required 2007-07-05 08:19:27 [MgmSrvr] INFO -- Node 2: President restarts arbitration thread [state=7] 2007-07-05 08:19:27 [MgmSrvr] ALERT -- Node 1: Node 3 Disconnected 2007-07-05 08:19:27 [MgmSrvr] ALERT -- Node 3: Forced node shutdown completed. Occured during startphase 5. Caused by error 2816: 'Read underflow(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'. 2007-07-05 08:19:27 [MgmSrvr] ALERT -- Node 2: Arbitration won - positive reply from node 1 2007-07-05 08:19:27 [MgmSrvr] INFO -- Node 2: DICT: remove lock by failed node 3 for NodeRestart 2007-07-05 08:19:27 [MgmSrvr] INFO -- Node 2: DICT: lock bs: 0 ops: 0 poll: 0 cnt: 0 queue: 2007-07-05 08:19:28 [MgmSrvr] INFO -- Node 2: Started arbitrator node 1 [ticket=179a00059689036c] 2007-07-05 08:19:31 [MgmSrvr] INFO -- Node 2: Communication to Node 3 opened 2007-07-05 08:21:53 [MgmSrvr] ALERT -- Node 2: Node 4 Disconnected 2007-07-05 08:21:53 [MgmSrvr] INFO -- Node 2: Communication to Node 4 closed 2007-07-05 08:21:53 [MgmSrvr] INFO -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 0000000000000002. 2007-07-05 08:21:56 [MgmSrvr] INFO -- Node 2: Communication to Node 4 opened 2007-07-05 08:27:57 [MgmSrvr] INFO -- Node 2: Local checkpoint 4 started. Keep GCI = 1094 oldest restorable GCI = 3