2012-04-13 07:54:39 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:54:39 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:54:59 [MgmtSrvr] INFO -- Node 4: Data usage is 78%(284511 32K pages of total 360448) 2012-04-13 07:54:59 [MgmtSrvr] INFO -- Node 4: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:54:59 [MgmtSrvr] INFO -- Node 6: Data usage is 76%(277034 32K pages of total 360448) 2012-04-13 07:54:59 [MgmtSrvr] INFO -- Node 6: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:55:08 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:55:08 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:55:09 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:55:09 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:55:10 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 8 seconds (max lag: 221) 2012-04-13 07:55:20 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 18 seconds (max lag: 221) 2012-04-13 07:55:29 [MgmtSrvr] INFO -- Node 4: Data usage is 78%(284511 32K pages of total 360448) 2012-04-13 07:55:29 [MgmtSrvr] INFO -- Node 4: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:55:30 [MgmtSrvr] WARNING -- Node 3: Node 23 missed heartbeat 2 2012-04-13 07:55:30 [MgmtSrvr] WARNING -- Node 3: Node 33 missed heartbeat 2 2012-04-13 07:55:30 [MgmtSrvr] WARNING -- Node 3: Node 34 missed heartbeat 2 2012-04-13 07:55:30 [MgmtSrvr] WARNING -- Node 3: Node 61 missed heartbeat 2 2012-04-13 07:55:30 [MgmtSrvr] WARNING -- Node 3: Node 67 missed heartbeat 2 2012-04-13 07:55:30 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 2 2012-04-13 07:55:30 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 28 seconds (max lag: 221) 2012-04-13 07:55:33 [MgmtSrvr] WARNING -- Node 5: Node 8 missed heartbeat 2 2012-04-13 07:55:33 [MgmtSrvr] WARNING -- Node 5: Node 24 missed heartbeat 2 2012-04-13 07:55:33 [MgmtSrvr] WARNING -- Node 5: Node 31 missed heartbeat 2 2012-04-13 07:55:33 [MgmtSrvr] WARNING -- Node 5: Node 48 missed heartbeat 2 2012-04-13 07:55:33 [MgmtSrvr] WARNING -- Node 5: Node 51 missed heartbeat 2 2012-04-13 07:55:33 [MgmtSrvr] WARNING -- Node 5: Node 70 missed heartbeat 2 2012-04-13 07:55:33 [MgmtSrvr] WARNING -- Node 5: Node 71 missed heartbeat 2 2012-04-13 07:55:33 [MgmtSrvr] WARNING -- Node 5: Node 77 missed heartbeat 2 2012-04-13 07:55:33 [MgmtSrvr] WARNING -- Node 5: Node 82 missed heartbeat 2 2012-04-13 07:55:38 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:55:38 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:55:39 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:55:39 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:55:40 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 38 seconds (max lag: 221) 2012-04-13 07:55:45 [MgmtSrvr] WARNING -- Node 3: Node 23 missed heartbeat 3 2012-04-13 07:55:45 [MgmtSrvr] WARNING -- Node 3: Node 33 missed heartbeat 3 2012-04-13 07:55:45 [MgmtSrvr] WARNING -- Node 3: Node 34 missed heartbeat 3 2012-04-13 07:55:45 [MgmtSrvr] WARNING -- Node 3: Node 54 missed heartbeat 2 2012-04-13 07:55:45 [MgmtSrvr] WARNING -- Node 3: Node 56 missed heartbeat 2 2012-04-13 07:55:45 [MgmtSrvr] WARNING -- Node 3: Node 61 missed heartbeat 3 2012-04-13 07:55:45 [MgmtSrvr] WARNING -- Node 3: Node 67 missed heartbeat 3 2012-04-13 07:55:45 [MgmtSrvr] WARNING -- Node 3: Node 76 missed heartbeat 2 2012-04-13 07:55:45 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 3 2012-04-13 07:55:46 [MgmtSrvr] WARNING -- Node 3: Node 6 missed heartbeat 2 2012-04-13 07:55:48 [MgmtSrvr] WARNING -- Node 5: Node 8 missed heartbeat 3 2012-04-13 07:55:48 [MgmtSrvr] WARNING -- Node 5: Node 24 missed heartbeat 3 2012-04-13 07:55:48 [MgmtSrvr] WARNING -- Node 5: Node 31 missed heartbeat 3 2012-04-13 07:55:48 [MgmtSrvr] WARNING -- Node 5: Node 48 missed heartbeat 3 2012-04-13 07:55:48 [MgmtSrvr] WARNING -- Node 5: Node 51 missed heartbeat 3 2012-04-13 07:55:48 [MgmtSrvr] WARNING -- Node 5: Node 70 missed heartbeat 3 2012-04-13 07:55:48 [MgmtSrvr] WARNING -- Node 5: Node 71 missed heartbeat 3 2012-04-13 07:55:48 [MgmtSrvr] WARNING -- Node 5: Node 77 missed heartbeat 3 2012-04-13 07:55:48 [MgmtSrvr] WARNING -- Node 5: Node 82 missed heartbeat 3 2012-04-13 07:55:50 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 48 seconds (max lag: 221) 2012-04-13 07:55:56 [MgmtSrvr] WARNING -- Node 4: Node 15 missed heartbeat 2 2012-04-13 07:55:56 [MgmtSrvr] WARNING -- Node 4: Node 16 missed heartbeat 2 2012-04-13 07:55:56 [MgmtSrvr] WARNING -- Node 4: Node 20 missed heartbeat 2 2012-04-13 07:55:58 [MgmtSrvr] ALERT -- Node 4: Node 15 Disconnected 2012-04-13 07:55:58 [MgmtSrvr] ALERT -- Node 4: Node 16 Disconnected 2012-04-13 07:55:58 [MgmtSrvr] ALERT -- Node 4: Node 20 Disconnected 2012-04-13 07:55:58 [MgmtSrvr] INFO -- Node 4: Communication to Node 15 closed 2012-04-13 07:55:58 [MgmtSrvr] INFO -- Node 4: Communication to Node 16 closed 2012-04-13 07:55:58 [MgmtSrvr] INFO -- Node 4: Communication to Node 20 closed 2012-04-13 07:55:58 [MgmtSrvr] INFO -- Node 5: Communication to Node 15 closed 2012-04-13 07:55:58 [MgmtSrvr] INFO -- Node 5: Communication to Node 16 closed 2012-04-13 07:55:58 [MgmtSrvr] INFO -- Node 5: Communication to Node 20 closed 2012-04-13 07:55:58 [MgmtSrvr] INFO -- Node 3: Communication to Node 15 closed 2012-04-13 07:55:58 [MgmtSrvr] INFO -- Node 3: Communication to Node 16 closed 2012-04-13 07:55:58 [MgmtSrvr] INFO -- Node 3: Communication to Node 20 closed 2012-04-13 07:55:58 [MgmtSrvr] ALERT -- Node 5: Node 15 Disconnected 2012-04-13 07:55:58 [MgmtSrvr] ALERT -- Node 5: Node 16 Disconnected 2012-04-13 07:55:58 [MgmtSrvr] ALERT -- Node 5: Node 20 Disconnected 2012-04-13 07:55:58 [MgmtSrvr] ALERT -- Node 3: Node 15 Disconnected 2012-04-13 07:55:58 [MgmtSrvr] ALERT -- Node 3: Node 16 Disconnected 2012-04-13 07:55:58 [MgmtSrvr] ALERT -- Node 3: Node 20 Disconnected 2012-04-13 07:55:59 [MgmtSrvr] INFO -- Node 4: Data usage is 78%(284511 32K pages of total 360448) 2012-04-13 07:55:59 [MgmtSrvr] INFO -- Node 4: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: Node 23 missed heartbeat 4 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 23 declared dead due to missed heartbeat 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: Node 33 missed heartbeat 4 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 33 declared dead due to missed heartbeat 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: Node 34 missed heartbeat 4 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 34 declared dead due to missed heartbeat 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: Node 54 missed heartbeat 3 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: Node 56 missed heartbeat 3 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: Node 61 missed heartbeat 4 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 61 declared dead due to missed heartbeat 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: Node 67 missed heartbeat 4 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 67 declared dead due to missed heartbeat 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: Node 76 missed heartbeat 3 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 4 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 79 declared dead due to missed heartbeat 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 3: Communication to Node 23 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 3: Communication to Node 33 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 3: Communication to Node 34 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 3: Communication to Node 61 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 3: Communication to Node 67 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 3: Communication to Node 79 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 5: Communication to Node 23 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 5: Communication to Node 33 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 5: Communication to Node 34 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 5: Communication to Node 61 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 5: Communication to Node 67 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 5: Communication to Node 79 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 4: Communication to Node 23 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 4: Communication to Node 33 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 4: Communication to Node 34 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 4: Communication to Node 61 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 4: Communication to Node 67 closed 2012-04-13 07:56:00 [MgmtSrvr] INFO -- Node 4: Communication to Node 79 closed 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 5: Node 23 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 5: Node 33 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 5: Node 34 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 5: Node 61 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 5: Node 67 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 5: Node 79 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 23 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 33 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 34 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 61 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 67 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 3: Node 79 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 4: Node 23 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 4: Node 33 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 4: Node 34 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 4: Node 61 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 4: Node 67 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] ALERT -- Node 4: Node 79 Disconnected 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_SAVE lag 60 seconds (max lag: 311s) 2012-04-13 07:56:00 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 58 seconds (max lag: 221) 2012-04-13 07:56:01 [MgmtSrvr] WARNING -- Node 3: Node 6 missed heartbeat 3 2012-04-13 07:56:01 [MgmtSrvr] ALERT -- Node 1: Node 6 Disconnected 2012-04-13 07:56:01 [MgmtSrvr] ALERT -- Node 1: Node 5 Disconnected 2012-04-13 07:56:02 [MgmtSrvr] ALERT -- Node 1: Node 3 Disconnected 2012-04-13 07:56:02 [MgmtSrvr] ALERT -- Node 1: Node 4 Disconnected 2012-04-13 07:56:06 [MgmtSrvr] INFO -- Node 1: Node 5 Connected 2012-04-13 07:56:06 [MgmtSrvr] INFO -- Node 1: Node 6 Connected 2012-04-13 07:56:06 [MgmtSrvr] INFO -- Node 1: Node 3 Connected 2012-04-13 07:56:08 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:56:08 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 5: Communication to Node 8 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 5: Communication to Node 24 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 5: Communication to Node 31 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 5: Communication to Node 48 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 5: Communication to Node 51 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 5: Communication to Node 70 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 5: Communication to Node 71 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 5: Communication to Node 77 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 5: Communication to Node 82 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 6: Communication to Node 8 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 6: Communication to Node 24 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 6: Communication to Node 31 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 6: Communication to Node 48 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 6: Communication to Node 51 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 6: Communication to Node 71 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 6: Communication to Node 77 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 6: Communication to Node 82 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Communication to Node 8 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Communication to Node 24 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Communication to Node 31 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Communication to Node 48 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Communication to Node 51 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Communication to Node 70 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Communication to Node 71 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Communication to Node 77 opened 2012-04-13 07:56:09 [MgmtSrvr] INFO -- Node 3: Communication to Node 82 opened 2012-04-13 07:56:10 [MgmtSrvr] INFO -- Node 6: Node 51 Connected 2012-04-13 07:56:10 [MgmtSrvr] INFO -- Node 5: Node 51 Connected 2012-04-13 07:56:10 [MgmtSrvr] INFO -- Node 5: Node 51: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:10 [MgmtSrvr] INFO -- Node 6: Node 51: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:10 [MgmtSrvr] INFO -- Node 3: Node 51 Connected 2012-04-13 07:56:10 [MgmtSrvr] INFO -- Node 3: Node 51: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:10 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 68 seconds (max lag: 221) 2012-04-13 07:56:12 [MgmtSrvr] INFO -- Node 3: Node 79 Connected 2012-04-13 07:56:15 [MgmtSrvr] WARNING -- Node 3: Node 54 missed heartbeat 4 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 3: Node 54 declared dead due to missed heartbeat 2012-04-13 07:56:15 [MgmtSrvr] WARNING -- Node 3: Node 56 missed heartbeat 4 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 3: Node 56 declared dead due to missed heartbeat 2012-04-13 07:56:15 [MgmtSrvr] WARNING -- Node 3: Node 76 missed heartbeat 4 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 3: Node 76 declared dead due to missed heartbeat 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 3: Communication to Node 54 closed 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 3: Communication to Node 56 closed 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 3: Communication to Node 76 closed 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 5: Communication to Node 54 closed 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 5: Communication to Node 56 closed 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 5: Communication to Node 76 closed 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 6: Communication to Node 54 closed 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 6: Communication to Node 56 closed 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 6: Communication to Node 76 closed 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 5: Node 54 Disconnected 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 5: Node 56 Disconnected 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 5: Node 76 Disconnected 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 3: Node 54 Disconnected 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 3: Node 56 Disconnected 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 3: Node 76 Disconnected 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 6: Node 54 Disconnected 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 6: Node 56 Disconnected 2012-04-13 07:56:15 [MgmtSrvr] ALERT -- Node 6: Node 76 Disconnected 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 6: Node 48 Connected 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 5: Node 48 Connected 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 5: Node 48: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 3: Node 48 Connected 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 6: Node 48: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:15 [MgmtSrvr] INFO -- Node 3: Node 48: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 5: Node 71 Connected 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 6: Node 71 Connected 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 5: Node 71: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 6: Node 71: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 3: Node 71 Connected 2012-04-13 07:56:16 [MgmtSrvr] WARNING -- Node 3: Node 6 missed heartbeat 4 2012-04-13 07:56:16 [MgmtSrvr] ALERT -- Node 3: Node 6 declared dead due to missed heartbeat 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 3: Communication to Node 6 closed 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 closed 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 3: Node 71: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:16 [MgmtSrvr] ALERT -- Node 5: Node 6 Disconnected 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 closed 2012-04-13 07:56:16 [MgmtSrvr] ALERT -- Node 3: Node 6 Disconnected 2012-04-13 07:56:16 [MgmtSrvr] INFO -- Node 3: Communication to Node 6 closed 2012-04-13 07:56:17 [MgmtSrvr] ALERT -- Node 6: Forced node shutdown completed. Caused by error 2315: 'Node declared dead. See error log for details(Arbitration error). Temporary error, restart node'. 2012-04-13 07:56:17 [MgmtSrvr] ALERT -- Node 1: Node 6 Disconnected 2012-04-13 07:56:17 [MgmtSrvr] INFO -- Node 3: Node 23 Connected 2012-04-13 07:56:17 [MgmtSrvr] INFO -- Node 5: Node 23 Connected 2012-04-13 07:56:17 [MgmtSrvr] INFO -- Node 3: Node 23: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:17 [MgmtSrvr] INFO -- Node 5: Node 23: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 5: Communication to Node 54 opened 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 5: Communication to Node 56 opened 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 5: Communication to Node 76 opened 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 5: Node 54 Connected 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 5: Node 54: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 54 opened 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 56 opened 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 76 opened 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 5: Node 24 Connected 2012-04-13 07:56:18 [MgmtSrvr] INFO -- Node 5: Node 31 Connected 2012-04-13 07:56:19 [MgmtSrvr] INFO -- Node 5: Node 24: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:19 [MgmtSrvr] INFO -- Node 5: Node 31: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:19 [MgmtSrvr] INFO -- Node 3: Communication to Node 6 opened 2012-04-13 07:56:20 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 opened 2012-04-13 07:56:20 [MgmtSrvr] INFO -- Node 3: Node 67 Connected 2012-04-13 07:56:20 [MgmtSrvr] INFO -- Node 5: Node 77 Connected 2012-04-13 07:56:20 [MgmtSrvr] INFO -- Node 3: Node 67: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:20 [MgmtSrvr] INFO -- Node 5: Node 67 Connected 2012-04-13 07:56:20 [MgmtSrvr] INFO -- Node 5: Node 77: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:20 [MgmtSrvr] INFO -- Node 5: Node 67: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:20 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 78 seconds (max lag: 221) 2012-04-13 07:56:20 [MgmtSrvr] INFO -- Node 3: Node 77 Connected 2012-04-13 07:56:20 [MgmtSrvr] INFO -- Node 3: Node 77: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:22 [MgmtSrvr] INFO -- Node 3: Node 76 Connected 2012-04-13 07:56:22 [MgmtSrvr] INFO -- Node 5: Node 76 Connected 2012-04-13 07:56:22 [MgmtSrvr] INFO -- Node 5: Node 76: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:22 [MgmtSrvr] INFO -- Node 3: Node 76: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:25 [MgmtSrvr] INFO -- Node 5: Node 34 Connected 2012-04-13 07:56:25 [MgmtSrvr] INFO -- Node 5: Node 34: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:28 [MgmtSrvr] INFO -- Node 5: Node 16 Connected 2012-04-13 07:56:28 [MgmtSrvr] INFO -- Node 5: Node 16: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:29 [MgmtSrvr] INFO -- Node 3: Node 20 Connected 2012-04-13 07:56:29 [MgmtSrvr] INFO -- Node 5: Node 20 Connected 2012-04-13 07:56:29 [MgmtSrvr] INFO -- Node 3: Node 15 Connected 2012-04-13 07:56:29 [MgmtSrvr] INFO -- Node 5: Node 15 Connected 2012-04-13 07:56:29 [MgmtSrvr] INFO -- Node 5: Node 20: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:29 [MgmtSrvr] INFO -- Node 3: Node 20: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:29 [MgmtSrvr] INFO -- Node 3: Node 15: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:29 [MgmtSrvr] INFO -- Node 5: Node 15: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:30 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 88 seconds (max lag: 221) 2012-04-13 07:56:35 [MgmtSrvr] INFO -- Node 5: Node 61 Connected 2012-04-13 07:56:35 [MgmtSrvr] INFO -- Node 5: Node 61: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:38 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:56:38 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 3: Node 82 Connected 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 5: Node 82 Connected 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 3: Node 56 Connected 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 5: Node 56 Connected 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 5: Node 82: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 3: Node 82: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 5: Node 56: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 3: Node 56: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 3: Node 54 Connected 2012-04-13 07:56:39 [MgmtSrvr] INFO -- Node 3: Node 54: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:40 [MgmtSrvr] INFO -- Node 3: Node 24 Connected 2012-04-13 07:56:40 [MgmtSrvr] INFO -- Node 3: Node 31 Connected 2012-04-13 07:56:40 [MgmtSrvr] INFO -- Node 3: Node 24: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:40 [MgmtSrvr] INFO -- Node 3: Node 31: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:40 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 98 seconds (max lag: 221) 2012-04-13 07:56:41 [MgmtSrvr] INFO -- Node 5: Node 33 Connected 2012-04-13 07:56:41 [MgmtSrvr] INFO -- Node 5: Node 33: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:41 [MgmtSrvr] INFO -- Node 3: Node 33 Connected 2012-04-13 07:56:41 [MgmtSrvr] INFO -- Node 3: Node 33: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:45 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 2 2012-04-13 07:56:50 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 108 seconds (max lag: 221) 2012-04-13 07:56:52 [MgmtSrvr] INFO -- Node 3: Node 8 Connected 2012-04-13 07:56:52 [MgmtSrvr] INFO -- Node 5: Node 8 Connected 2012-04-13 07:56:52 [MgmtSrvr] INFO -- Node 5: Node 8: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:52 [MgmtSrvr] INFO -- Node 3: Node 8: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:56:58 [MgmtSrvr] INFO -- Node 3: Node 16 Connected 2012-04-13 07:56:59 [MgmtSrvr] INFO -- Node 3: Node 16: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:00 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 3 2012-04-13 07:57:00 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_SAVE lag 120 seconds (max lag: 311s) 2012-04-13 07:57:00 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 118 seconds (max lag: 221) 2012-04-13 07:57:01 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 2 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 5: Node 70 Connected 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 5: Node 70: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 3: Node 70 Connected 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 3: Node 70: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:08 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:57:08 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:57:09 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:57:09 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:57:10 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 128 seconds (max lag: 221) 2012-04-13 07:57:15 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 4 2012-04-13 07:57:15 [MgmtSrvr] ALERT -- Node 3: Node 79 declared dead due to missed heartbeat 2012-04-13 07:57:15 [MgmtSrvr] INFO -- Node 3: Communication to Node 79 closed 2012-04-13 07:57:15 [MgmtSrvr] INFO -- Node 5: Communication to Node 79 closed 2012-04-13 07:57:15 [MgmtSrvr] ALERT -- Node 5: Node 79 Disconnected 2012-04-13 07:57:15 [MgmtSrvr] ALERT -- Node 3: Node 79 Disconnected 2012-04-13 07:57:16 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 3 2012-04-13 07:57:17 [MgmtSrvr] INFO -- Node 3: Node 61 Connected 2012-04-13 07:57:17 [MgmtSrvr] INFO -- Node 3: Node 61: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 79 opened 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 5: Communication to Node 79 opened 2012-04-13 07:57:20 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 138 seconds (max lag: 221) 2012-04-13 07:57:30 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 148 seconds (max lag: 221) 2012-04-13 07:57:31 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 4 2012-04-13 07:57:31 [MgmtSrvr] ALERT -- Node 3: Node 4 declared dead due to missed heartbeat 2012-04-13 07:57:31 [MgmtSrvr] INFO -- Node 3: Communication to Node 4 closed 2012-04-13 07:57:31 [MgmtSrvr] INFO -- Node 3: Communication to Node 6 closed 2012-04-13 07:57:31 [MgmtSrvr] INFO -- Node 5: Communication to Node 4 closed 2012-04-13 07:57:31 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 closed 2012-04-13 07:57:31 [MgmtSrvr] ALERT -- Node 5: Node 4 Disconnected 2012-04-13 07:57:31 [MgmtSrvr] ALERT -- Node 5: Node 6 Disconnected 2012-04-13 07:57:31 [MgmtSrvr] INFO -- Node 5: Communication to Node 4 closed 2012-04-13 07:57:31 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 closed 2012-04-13 07:57:32 [MgmtSrvr] ALERT -- Node 5: Node 3 Disconnected 2012-04-13 07:57:32 [MgmtSrvr] ALERT -- Node 4: Forced node shutdown completed. Caused by error 2315: 'Node declared dead. See error log for details(Arbitration error). Temporary error, restart node'. 2012-04-13 07:57:32 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'. 2012-04-13 07:57:32 [MgmtSrvr] ALERT -- Node 1: Node 3 Disconnected 2012-04-13 07:57:33 [MgmtSrvr] ALERT -- Node 5: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'. 2012-04-13 07:57:33 [MgmtSrvr] ALERT -- Node 1: Node 5 Disconnected