2012-04-13 07:56:02 [MgmtSrvr] INFO -- Node 4: Data usage is 78%(284511 32K pages of total 360448) 2012-04-13 07:56:02 [MgmtSrvr] INFO -- Node 4: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:56:02 [MgmtSrvr] INFO -- Node 6: Data usage is 76%(277034 32K pages of total 360448) 2012-04-13 07:56:02 [MgmtSrvr] INFO -- Node 6: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:56:11 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:56:11 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:56:12 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:56:12 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:56:13 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 8 seconds (max lag: 221) 2012-04-13 07:56:23 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 18 seconds (max lag: 221) 2012-04-13 07:56:32 [MgmtSrvr] INFO -- Node 4: Data usage is 78%(284511 32K pages of total 360448) 2012-04-13 07:56:32 [MgmtSrvr] INFO -- Node 4: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:56:32 [MgmtSrvr] INFO -- Node 6: Data usage is 76%(277034 32K pages of total 360448) 2012-04-13 07:56:32 [MgmtSrvr] INFO -- Node 6: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:56:33 [MgmtSrvr] WARNING -- Node 3: Node 23 missed heartbeat 2 2012-04-13 07:56:33 [MgmtSrvr] WARNING -- Node 3: Node 33 missed heartbeat 2 2012-04-13 07:56:33 [MgmtSrvr] WARNING -- Node 3: Node 34 missed heartbeat 2 2012-04-13 07:56:33 [MgmtSrvr] WARNING -- Node 3: Node 61 missed heartbeat 2 2012-04-13 07:56:33 [MgmtSrvr] WARNING -- Node 3: Node 67 missed heartbeat 2 2012-04-13 07:56:33 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 2 2012-04-13 07:56:33 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 28 seconds (max lag: 221) 2012-04-13 07:56:36 [MgmtSrvr] WARNING -- Node 5: Node 8 missed heartbeat 2 2012-04-13 07:56:36 [MgmtSrvr] WARNING -- Node 5: Node 24 missed heartbeat 2 2012-04-13 07:56:36 [MgmtSrvr] WARNING -- Node 5: Node 31 missed heartbeat 2 2012-04-13 07:56:36 [MgmtSrvr] WARNING -- Node 5: Node 48 missed heartbeat 2 2012-04-13 07:56:36 [MgmtSrvr] WARNING -- Node 5: Node 51 missed heartbeat 2 2012-04-13 07:56:36 [MgmtSrvr] WARNING -- Node 5: Node 70 missed heartbeat 2 2012-04-13 07:56:36 [MgmtSrvr] WARNING -- Node 5: Node 71 missed heartbeat 2 2012-04-13 07:56:36 [MgmtSrvr] WARNING -- Node 5: Node 77 missed heartbeat 2 2012-04-13 07:56:36 [MgmtSrvr] WARNING -- Node 5: Node 82 missed heartbeat 2 2012-04-13 07:56:41 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:56:41 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:56:42 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:56:42 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:56:43 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 38 seconds (max lag: 221) 2012-04-13 07:56:46 [MgmtSrvr] WARNING -- Node 6: Node 1 missed heartbeat 2 2012-04-13 07:56:48 [MgmtSrvr] WARNING -- Node 3: Node 23 missed heartbeat 3 2012-04-13 07:56:48 [MgmtSrvr] WARNING -- Node 3: Node 33 missed heartbeat 3 2012-04-13 07:56:48 [MgmtSrvr] WARNING -- Node 3: Node 34 missed heartbeat 3 2012-04-13 07:56:48 [MgmtSrvr] WARNING -- Node 3: Node 54 missed heartbeat 2 2012-04-13 07:56:48 [MgmtSrvr] WARNING -- Node 3: Node 56 missed heartbeat 2 2012-04-13 07:56:48 [MgmtSrvr] WARNING -- Node 3: Node 61 missed heartbeat 3 2012-04-13 07:56:48 [MgmtSrvr] WARNING -- Node 3: Node 67 missed heartbeat 3 2012-04-13 07:56:48 [MgmtSrvr] WARNING -- Node 3: Node 76 missed heartbeat 2 2012-04-13 07:56:48 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 3 2012-04-13 07:56:49 [MgmtSrvr] WARNING -- Node 3: Node 6 missed heartbeat 2 2012-04-13 07:56:51 [MgmtSrvr] WARNING -- Node 5: Node 8 missed heartbeat 3 2012-04-13 07:56:51 [MgmtSrvr] WARNING -- Node 5: Node 24 missed heartbeat 3 2012-04-13 07:56:51 [MgmtSrvr] WARNING -- Node 5: Node 31 missed heartbeat 3 2012-04-13 07:56:51 [MgmtSrvr] WARNING -- Node 5: Node 48 missed heartbeat 3 2012-04-13 07:56:51 [MgmtSrvr] WARNING -- Node 5: Node 51 missed heartbeat 3 2012-04-13 07:56:51 [MgmtSrvr] WARNING -- Node 5: Node 70 missed heartbeat 3 2012-04-13 07:56:51 [MgmtSrvr] WARNING -- Node 5: Node 71 missed heartbeat 3 2012-04-13 07:56:51 [MgmtSrvr] WARNING -- Node 5: Node 77 missed heartbeat 3 2012-04-13 07:56:51 [MgmtSrvr] WARNING -- Node 5: Node 82 missed heartbeat 3 2012-04-13 07:56:53 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 48 seconds (max lag: 221) 2012-04-13 07:57:00 [MgmtSrvr] WARNING -- Node 4: Node 15 missed heartbeat 2 2012-04-13 07:57:00 [MgmtSrvr] WARNING -- Node 4: Node 16 missed heartbeat 2 2012-04-13 07:57:00 [MgmtSrvr] WARNING -- Node 4: Node 20 missed heartbeat 2 2012-04-13 07:57:01 [MgmtSrvr] WARNING -- Node 6: Node 1 missed heartbeat 3 2012-04-13 07:57:01 [MgmtSrvr] ALERT -- Node 4: Node 15 Disconnected 2012-04-13 07:57:01 [MgmtSrvr] ALERT -- Node 4: Node 16 Disconnected 2012-04-13 07:57:01 [MgmtSrvr] ALERT -- Node 4: Node 20 Disconnected 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 4: Communication to Node 15 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 4: Communication to Node 16 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 4: Communication to Node 20 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 5: Communication to Node 15 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 5: Communication to Node 16 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 5: Communication to Node 20 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 6: Communication to Node 15 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 6: Communication to Node 16 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 6: Communication to Node 20 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 3: Communication to Node 15 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 3: Communication to Node 16 closed 2012-04-13 07:57:01 [MgmtSrvr] INFO -- Node 3: Communication to Node 20 closed 2012-04-13 07:57:01 [MgmtSrvr] ALERT -- Node 5: Node 15 Disconnected 2012-04-13 07:57:01 [MgmtSrvr] ALERT -- Node 5: Node 16 Disconnected 2012-04-13 07:57:01 [MgmtSrvr] ALERT -- Node 5: Node 20 Disconnected 2012-04-13 07:57:01 [MgmtSrvr] ALERT -- Node 6: Node 15 Disconnected 2012-04-13 07:57:01 [MgmtSrvr] ALERT -- Node 6: Node 16 Disconnected 2012-04-13 07:57:01 [MgmtSrvr] ALERT -- Node 6: Node 20 Disconnected 2012-04-13 07:57:02 [MgmtSrvr] ALERT -- Node 3: Node 15 Disconnected 2012-04-13 07:57:02 [MgmtSrvr] ALERT -- Node 3: Node 16 Disconnected 2012-04-13 07:57:02 [MgmtSrvr] ALERT -- Node 3: Node 20 Disconnected 2012-04-13 07:57:02 [MgmtSrvr] INFO -- Node 4: Data usage is 78%(284511 32K pages of total 360448) 2012-04-13 07:57:02 [MgmtSrvr] INFO -- Node 4: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:57:02 [MgmtSrvr] INFO -- Node 6: Data usage is 76%(277034 32K pages of total 360448) 2012-04-13 07:57:02 [MgmtSrvr] INFO -- Node 6: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:57:03 [MgmtSrvr] WARNING -- Node 3: Node 23 missed heartbeat 4 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 23 declared dead due to missed heartbeat 2012-04-13 07:57:03 [MgmtSrvr] WARNING -- Node 3: Node 33 missed heartbeat 4 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 33 declared dead due to missed heartbeat 2012-04-13 07:57:03 [MgmtSrvr] WARNING -- Node 3: Node 34 missed heartbeat 4 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 34 declared dead due to missed heartbeat 2012-04-13 07:57:03 [MgmtSrvr] WARNING -- Node 3: Node 54 missed heartbeat 3 2012-04-13 07:57:03 [MgmtSrvr] WARNING -- Node 3: Node 56 missed heartbeat 3 2012-04-13 07:57:03 [MgmtSrvr] WARNING -- Node 3: Node 61 missed heartbeat 4 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 61 declared dead due to missed heartbeat 2012-04-13 07:57:03 [MgmtSrvr] WARNING -- Node 3: Node 67 missed heartbeat 4 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 67 declared dead due to missed heartbeat 2012-04-13 07:57:03 [MgmtSrvr] WARNING -- Node 3: Node 76 missed heartbeat 3 2012-04-13 07:57:03 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 4 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 79 declared dead due to missed heartbeat 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 3: Communication to Node 23 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 3: Communication to Node 33 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 3: Communication to Node 34 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 3: Communication to Node 61 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 3: Communication to Node 67 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 3: Communication to Node 79 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 6: Communication to Node 23 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 6: Communication to Node 33 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 6: Communication to Node 34 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 6: Communication to Node 61 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 6: Communication to Node 67 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 6: Communication to Node 79 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 5: Communication to Node 23 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 5: Communication to Node 33 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 5: Communication to Node 34 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 5: Communication to Node 61 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 5: Communication to Node 67 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 5: Communication to Node 79 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 4: Communication to Node 23 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 4: Communication to Node 33 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 4: Communication to Node 34 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 4: Communication to Node 61 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 4: Communication to Node 67 closed 2012-04-13 07:57:03 [MgmtSrvr] INFO -- Node 4: Communication to Node 79 closed 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 6: Node 23 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 6: Node 33 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 6: Node 34 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 6: Node 61 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 6: Node 67 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 6: Node 79 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 5: Node 23 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 5: Node 33 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 5: Node 34 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 5: Node 61 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 5: Node 67 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 5: Node 79 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 23 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 33 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 34 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 61 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 67 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 3: Node 79 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 4: Node 23 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 4: Node 33 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 4: Node 34 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 4: Node 61 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 4: Node 67 Disconnected 2012-04-13 07:57:03 [MgmtSrvr] ALERT -- Node 4: Node 79 Disconnected 2012-04-13 07:57:04 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_SAVE lag 60 seconds (max lag: 311s) 2012-04-13 07:57:04 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 58 seconds (max lag: 221) 2012-04-13 07:57:04 [MgmtSrvr] WARNING -- Node 3: Node 6 missed heartbeat 3 2012-04-13 07:57:05 [MgmtSrvr] ALERT -- Node 6: Node 1 Disconnected 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 6: Communication to Node 1 closed 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 6: Lost arbitrator node 1 - process failure [state=6] 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 4: Communication to Node 1 closed 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 4: Lost arbitrator node 1 - process failure [state=6] 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 5: Communication to Node 1 closed 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 5: Lost arbitrator node 1 - process failure [state=6] 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 3: Communication to Node 1 closed 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 3: Lost arbitrator node 1 - process failure [state=6] 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 3: President restarts arbitration thread [state=1] 2012-04-13 07:57:05 [MgmtSrvr] ALERT -- Node 5: Node 1 Disconnected 2012-04-13 07:57:05 [MgmtSrvr] ALERT -- Node 4: Node 1 Disconnected 2012-04-13 07:57:05 [MgmtSrvr] ALERT -- Node 3: Node 1 Disconnected 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 4: Communication to Node 1 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 4: Communication to Node 15 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 4: Communication to Node 16 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 4: Communication to Node 20 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 5: Communication to Node 1 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 5: Communication to Node 15 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 5: Communication to Node 16 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 5: Communication to Node 20 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 6: Communication to Node 1 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 6: Communication to Node 15 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 6: Communication to Node 16 opened 2012-04-13 07:57:05 [MgmtSrvr] INFO -- Node 6: Communication to Node 20 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 1 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 15 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 16 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 20 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 23 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 33 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 34 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 61 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 67 opened 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 79 opened 2012-04-13 07:57:06 [MgmtSrvr] WARNING -- Node 5: Node 8 missed heartbeat 4 2012-04-13 07:57:06 [MgmtSrvr] ALERT -- Node 5: Node 8 declared dead due to missed heartbeat 2012-04-13 07:57:06 [MgmtSrvr] WARNING -- Node 5: Node 24 missed heartbeat 4 2012-04-13 07:57:06 [MgmtSrvr] ALERT -- Node 5: Node 24 declared dead due to missed heartbeat 2012-04-13 07:57:06 [MgmtSrvr] WARNING -- Node 5: Node 31 missed heartbeat 4 2012-04-13 07:57:06 [MgmtSrvr] ALERT -- Node 5: Node 31 declared dead due to missed heartbeat 2012-04-13 07:57:06 [MgmtSrvr] WARNING -- Node 5: Node 48 missed heartbeat 4 2012-04-13 07:57:06 [MgmtSrvr] ALERT -- Node 5: Node 48 declared dead due to missed heartbeat 2012-04-13 07:57:06 [MgmtSrvr] WARNING -- Node 5: Node 51 missed heartbeat 4 2012-04-13 07:57:06 [MgmtSrvr] ALERT -- Node 5: Node 51 declared dead due to missed heartbeat 2012-04-13 07:57:06 [MgmtSrvr] WARNING -- Node 5: Node 70 missed heartbeat 4 2012-04-13 07:57:06 [MgmtSrvr] ALERT -- Node 5: Node 70 declared dead due to missed heartbeat 2012-04-13 07:57:06 [MgmtSrvr] WARNING -- Node 5: Node 71 missed heartbeat 4 2012-04-13 07:57:06 [MgmtSrvr] ALERT -- Node 5: Node 71 declared dead due to missed heartbeat 2012-04-13 07:57:06 [MgmtSrvr] WARNING -- Node 5: Node 77 missed heartbeat 4 2012-04-13 07:57:06 [MgmtSrvr] ALERT -- Node 5: Node 77 declared dead due to missed heartbeat 2012-04-13 07:57:06 [MgmtSrvr] WARNING -- Node 5: Node 82 missed heartbeat 4 2012-04-13 07:57:06 [MgmtSrvr] ALERT -- Node 5: Node 82 declared dead due to missed heartbeat 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 4: Communication to Node 8 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 4: Communication to Node 24 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 4: Communication to Node 31 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 4: Communication to Node 48 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 4: Communication to Node 51 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 4: Communication to Node 70 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 4: Communication to Node 71 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 4: Communication to Node 77 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 4: Communication to Node 82 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 5: Communication to Node 8 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 5: Communication to Node 24 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 5: Communication to Node 31 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 5: Communication to Node 48 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 5: Communication to Node 51 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 5: Communication to Node 70 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 5: Communication to Node 71 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 5: Communication to Node 77 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 5: Communication to Node 82 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 8 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 24 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 31 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 48 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 51 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 70 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 71 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 77 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 6: Communication to Node 82 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 8 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 24 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 31 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 48 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 51 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 70 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 71 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 77 closed 2012-04-13 07:57:06 [MgmtSrvr] INFO -- Node 3: Communication to Node 82 closed 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 6: Node 34 Connected 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 3: Communication to Node 23 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 3: Communication to Node 33 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 3: Communication to Node 34 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 3: Communication to Node 61 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 3: Communication to Node 67 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 3: Communication to Node 79 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 3: Node 34 Connected 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 6: Node 34: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 3: Node 34: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 4: Communication to Node 23 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 4: Communication to Node 33 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 4: Communication to Node 34 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 4: Communication to Node 61 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 4: Communication to Node 67 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 4: Communication to Node 79 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 5: Communication to Node 23 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 5: Communication to Node 33 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 5: Communication to Node 34 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 5: Communication to Node 61 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 5: Communication to Node 67 opened 2012-04-13 07:57:07 [MgmtSrvr] INFO -- Node 5: Communication to Node 79 opened 2012-04-13 07:57:09 [MgmtSrvr] INFO -- Node 5: Node 1 Connected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 5: Node 8 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 5: Node 24 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 5: Node 31 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 5: Node 48 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 5: Node 51 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 5: Node 70 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 5: Node 71 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 5: Node 77 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 5: Node 82 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] INFO -- Node 5: Node 1: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:09 [MgmtSrvr] INFO -- Node 6: Node 1 Connected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 6: Node 8 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 6: Node 24 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 6: Node 31 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 6: Node 48 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 6: Node 51 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 6: Node 70 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 6: Node 71 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 6: Node 77 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] ALERT -- Node 6: Node 82 Disconnected 2012-04-13 07:57:09 [MgmtSrvr] INFO -- Node 6: Node 1: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:10 [MgmtSrvr] INFO -- Node 3: Node 1 Connected 2012-04-13 07:57:10 [MgmtSrvr] ALERT -- Node 3: Node 8 Disconnected 2012-04-13 07:57:10 [MgmtSrvr] ALERT -- Node 3: Node 24 Disconnected 2012-04-13 07:57:10 [MgmtSrvr] ALERT -- Node 3: Node 31 Disconnected 2012-04-13 07:57:10 [MgmtSrvr] ALERT -- Node 3: Node 48 Disconnected 2012-04-13 07:57:10 [MgmtSrvr] ALERT -- Node 3: Node 51 Disconnected 2012-04-13 07:57:10 [MgmtSrvr] ALERT -- Node 3: Node 70 Disconnected 2012-04-13 07:57:10 [MgmtSrvr] ALERT -- Node 3: Node 71 Disconnected 2012-04-13 07:57:10 [MgmtSrvr] ALERT -- Node 3: Node 77 Disconnected 2012-04-13 07:57:10 [MgmtSrvr] ALERT -- Node 3: Node 82 Disconnected 2012-04-13 07:57:10 [MgmtSrvr] INFO -- Node 3: Node 1: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:11 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:57:11 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 5: Communication to Node 8 opened 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 5: Communication to Node 24 opened 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 5: Communication to Node 31 opened 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 5: Communication to Node 48 opened 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 5: Communication to Node 51 opened 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 5: Communication to Node 70 opened 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 5: Communication to Node 71 opened 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 5: Communication to Node 77 opened 2012-04-13 07:57:12 [MgmtSrvr] INFO -- Node 5: Communication to Node 82 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Communication to Node 8 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Communication to Node 24 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Communication to Node 31 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Communication to Node 48 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Communication to Node 51 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Communication to Node 71 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Communication to Node 77 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Communication to Node 82 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Communication to Node 8 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Communication to Node 24 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Communication to Node 31 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Communication to Node 48 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Communication to Node 51 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Communication to Node 70 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Communication to Node 71 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Communication to Node 77 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Communication to Node 82 opened 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Node 51 Connected 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 5: Node 51 Connected 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 5: Node 51: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 6: Node 51: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:13 [MgmtSrvr] INFO -- Node 3: Node 51 Connected 2012-04-13 07:57:14 [MgmtSrvr] INFO -- Node 3: Node 51: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:14 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 68 seconds (max lag: 221) 2012-04-13 07:57:15 [MgmtSrvr] INFO -- Node 3: Node 79 Connected 2012-04-13 07:57:18 [MgmtSrvr] WARNING -- Node 3: Node 54 missed heartbeat 4 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 3: Node 54 declared dead due to missed heartbeat 2012-04-13 07:57:18 [MgmtSrvr] WARNING -- Node 3: Node 56 missed heartbeat 4 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 3: Node 56 declared dead due to missed heartbeat 2012-04-13 07:57:18 [MgmtSrvr] WARNING -- Node 3: Node 76 missed heartbeat 4 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 3: Node 76 declared dead due to missed heartbeat 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 54 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 56 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 76 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 4: Communication to Node 54 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 4: Communication to Node 56 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 4: Communication to Node 76 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 5: Communication to Node 54 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 5: Communication to Node 56 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 5: Communication to Node 76 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 6: Communication to Node 54 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 6: Communication to Node 56 closed 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 6: Communication to Node 76 closed 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 5: Node 54 Disconnected 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 5: Node 56 Disconnected 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 5: Node 76 Disconnected 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 3: Node 54 Disconnected 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 3: Node 56 Disconnected 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 3: Node 76 Disconnected 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 6: Node 54 Disconnected 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 6: Node 56 Disconnected 2012-04-13 07:57:18 [MgmtSrvr] ALERT -- Node 6: Node 76 Disconnected 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 6: Node 48 Connected 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 5: Node 48 Connected 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 5: Node 48: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 3: Node 48 Connected 2012-04-13 07:57:18 [MgmtSrvr] INFO -- Node 6: Node 48: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:19 [MgmtSrvr] INFO -- Node 3: Node 48: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:19 [MgmtSrvr] INFO -- Node 5: Node 71 Connected 2012-04-13 07:57:19 [MgmtSrvr] INFO -- Node 6: Node 71 Connected 2012-04-13 07:57:19 [MgmtSrvr] INFO -- Node 5: Node 71: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:19 [MgmtSrvr] INFO -- Node 6: Node 71: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:19 [MgmtSrvr] INFO -- Node 3: Node 71 Connected 2012-04-13 07:57:19 [MgmtSrvr] WARNING -- Node 3: Node 6 missed heartbeat 4 2012-04-13 07:57:19 [MgmtSrvr] ALERT -- Node 3: Node 6 declared dead due to missed heartbeat 2012-04-13 07:57:19 [MgmtSrvr] INFO -- Node 3: Communication to Node 6 closed 2012-04-13 07:57:19 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 closed 2012-04-13 07:57:19 [MgmtSrvr] INFO -- Node 3: Node 71: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:20 [MgmtSrvr] ALERT -- Node 5: Node 6 Disconnected 2012-04-13 07:57:20 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 closed 2012-04-13 07:57:20 [MgmtSrvr] ALERT -- Node 3: Node 6 Disconnected 2012-04-13 07:57:20 [MgmtSrvr] INFO -- Node 3: Communication to Node 6 closed 2012-04-13 07:57:21 [MgmtSrvr] ALERT -- Node 2: Node 6 Disconnected 2012-04-13 07:57:21 [MgmtSrvr] ALERT -- Node 6: Forced node shutdown completed. Caused by error 2315: 'Node declared dead. See error log for details(Arbitration error). Temporary error, restart node'. 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 3: Node 23 Connected 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 4: Node 23 Connected 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 5: Node 23 Connected 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 3: Node 23: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 4: Node 23: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 5: Node 23: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 5: Communication to Node 54 opened 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 5: Communication to Node 56 opened 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 5: Communication to Node 76 opened 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 5: Node 54 Connected 2012-04-13 07:57:21 [MgmtSrvr] INFO -- Node 5: Node 54: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:22 [MgmtSrvr] INFO -- Node 3: Communication to Node 54 opened 2012-04-13 07:57:22 [MgmtSrvr] INFO -- Node 3: Communication to Node 56 opened 2012-04-13 07:57:22 [MgmtSrvr] INFO -- Node 3: Communication to Node 76 opened 2012-04-13 07:57:22 [MgmtSrvr] INFO -- Node 5: Node 24 Connected 2012-04-13 07:57:22 [MgmtSrvr] INFO -- Node 5: Node 31 Connected 2012-04-13 07:57:22 [MgmtSrvr] INFO -- Node 5: Node 24: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:22 [MgmtSrvr] INFO -- Node 5: Node 31: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:23 [MgmtSrvr] INFO -- Node 3: Communication to Node 6 opened 2012-04-13 07:57:23 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 opened 2012-04-13 07:57:23 [MgmtSrvr] INFO -- Node 3: Node 67 Connected 2012-04-13 07:57:23 [MgmtSrvr] INFO -- Node 5: Node 77 Connected 2012-04-13 07:57:23 [MgmtSrvr] INFO -- Node 3: Node 67: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:23 [MgmtSrvr] INFO -- Node 4: Node 67 Connected 2012-04-13 07:57:23 [MgmtSrvr] INFO -- Node 5: Node 67 Connected 2012-04-13 07:57:24 [MgmtSrvr] INFO -- Node 5: Node 77: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:24 [MgmtSrvr] INFO -- Node 4: Node 67: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:24 [MgmtSrvr] INFO -- Node 5: Node 67: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:24 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 78 seconds (max lag: 221) 2012-04-13 07:57:24 [MgmtSrvr] INFO -- Node 3: Node 77 Connected 2012-04-13 07:57:24 [MgmtSrvr] INFO -- Node 3: Node 77: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:26 [MgmtSrvr] INFO -- Node 3: Node 76 Connected 2012-04-13 07:57:26 [MgmtSrvr] INFO -- Node 5: Node 76 Connected 2012-04-13 07:57:26 [MgmtSrvr] INFO -- Node 5: Node 76: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:26 [MgmtSrvr] INFO -- Node 3: Node 76: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:28 [MgmtSrvr] INFO -- Node 4: Node 34 Connected 2012-04-13 07:57:28 [MgmtSrvr] INFO -- Node 5: Node 34 Connected 2012-04-13 07:57:28 [MgmtSrvr] INFO -- Node 4: Node 34: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:28 [MgmtSrvr] INFO -- Node 5: Node 34: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:32 [MgmtSrvr] INFO -- Node 5: Node 16 Connected 2012-04-13 07:57:32 [MgmtSrvr] INFO -- Node 5: Node 16: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:32 [MgmtSrvr] INFO -- Node 4: Data usage is 78%(284511 32K pages of total 360448) 2012-04-13 07:57:32 [MgmtSrvr] INFO -- Node 4: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:57:32 [MgmtSrvr] INFO -- Node 4: Node 20 Connected 2012-04-13 07:57:32 [MgmtSrvr] INFO -- Node 3: Node 20 Connected 2012-04-13 07:57:32 [MgmtSrvr] INFO -- Node 5: Node 20 Connected 2012-04-13 07:57:33 [MgmtSrvr] INFO -- Node 4: Node 15 Connected 2012-04-13 07:57:33 [MgmtSrvr] INFO -- Node 3: Node 15 Connected 2012-04-13 07:57:33 [MgmtSrvr] INFO -- Node 5: Node 15 Connected 2012-04-13 07:57:33 [MgmtSrvr] INFO -- Node 4: Node 20: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:33 [MgmtSrvr] INFO -- Node 5: Node 20: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:33 [MgmtSrvr] INFO -- Node 3: Node 20: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:33 [MgmtSrvr] INFO -- Node 4: Node 15: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:33 [MgmtSrvr] INFO -- Node 5: Node 15: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:33 [MgmtSrvr] INFO -- Node 3: Node 15: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:34 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 88 seconds (max lag: 221) 2012-04-13 07:57:38 [MgmtSrvr] INFO -- Node 4: Node 61 Connected 2012-04-13 07:57:38 [MgmtSrvr] INFO -- Node 5: Node 61 Connected 2012-04-13 07:57:38 [MgmtSrvr] INFO -- Node 4: Node 61: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:38 [MgmtSrvr] INFO -- Node 5: Node 61: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:41 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:57:41 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 3: Node 82 Connected 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 5: Node 82 Connected 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 3: Node 56 Connected 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 5: Node 56 Connected 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 5: Node 82: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 3: Node 82: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 5: Node 56: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:42 [MgmtSrvr] INFO -- Node 3: Node 56: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:43 [MgmtSrvr] INFO -- Node 3: Node 54 Connected 2012-04-13 07:57:43 [MgmtSrvr] INFO -- Node 3: Node 54: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:43 [MgmtSrvr] INFO -- Node 3: Node 24 Connected 2012-04-13 07:57:43 [MgmtSrvr] INFO -- Node 3: Node 31 Connected 2012-04-13 07:57:43 [MgmtSrvr] INFO -- Node 3: Node 24: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:43 [MgmtSrvr] INFO -- Node 3: Node 31: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:44 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 98 seconds (max lag: 221) 2012-04-13 07:57:44 [MgmtSrvr] INFO -- Node 4: Node 33 Connected 2012-04-13 07:57:44 [MgmtSrvr] INFO -- Node 5: Node 33 Connected 2012-04-13 07:57:44 [MgmtSrvr] INFO -- Node 4: Node 33: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:44 [MgmtSrvr] INFO -- Node 5: Node 33: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:44 [MgmtSrvr] INFO -- Node 3: Node 33 Connected 2012-04-13 07:57:44 [MgmtSrvr] INFO -- Node 3: Node 33: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:48 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 2 2012-04-13 07:57:54 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 108 seconds (max lag: 221) 2012-04-13 07:57:55 [MgmtSrvr] INFO -- Node 3: Node 8 Connected 2012-04-13 07:57:55 [MgmtSrvr] INFO -- Node 5: Node 8 Connected 2012-04-13 07:57:56 [MgmtSrvr] INFO -- Node 5: Node 8: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:57:56 [MgmtSrvr] INFO -- Node 3: Node 8: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:02 [MgmtSrvr] INFO -- Node 4: Node 16 Connected 2012-04-13 07:58:02 [MgmtSrvr] INFO -- Node 3: Node 16 Connected 2012-04-13 07:58:02 [MgmtSrvr] INFO -- Node 4: Node 16: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:02 [MgmtSrvr] INFO -- Node 3: Node 16: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:02 [MgmtSrvr] INFO -- Node 4: Data usage is 78%(284511 32K pages of total 360448) 2012-04-13 07:58:02 [MgmtSrvr] INFO -- Node 4: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:58:03 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 3 2012-04-13 07:58:04 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_SAVE lag 120 seconds (max lag: 311s) 2012-04-13 07:58:04 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 118 seconds (max lag: 221) 2012-04-13 07:58:04 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 2 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 6 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] INFO -- Node 4: Communication to Node 6 closed 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 8 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 24 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 31 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 48 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 51 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 54 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 56 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 70 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 71 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 76 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 77 Disconnected 2012-04-13 07:58:05 [MgmtSrvr] ALERT -- Node 4: Node 82 Disconnected 2012-04-13 07:58:06 [MgmtSrvr] INFO -- Node 5: Node 70 Connected 2012-04-13 07:58:07 [MgmtSrvr] INFO -- Node 5: Node 70: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:07 [MgmtSrvr] INFO -- Node 3: Node 70 Connected 2012-04-13 07:58:07 [MgmtSrvr] INFO -- Node 3: Node 70: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 6 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 8 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 24 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 31 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 48 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 51 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 54 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 56 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 70 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 71 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 76 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 77 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Communication to Node 82 opened 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 56 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 71 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 54 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 56: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 48 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 77 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 54: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 82 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 70 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 76 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 48: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 71: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 70: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 82: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 31 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 51 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 8 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 77: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 24 Connected 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 76: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 31: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 24: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 51: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:09 [MgmtSrvr] INFO -- Node 4: Node 8: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:11 [MgmtSrvr] INFO -- Node 5: Data usage is 76%(277019 32K pages of total 360448) 2012-04-13 07:58:11 [MgmtSrvr] INFO -- Node 5: Index usage is 23%(30338 8K pages of total 131200) 2012-04-13 07:58:12 [MgmtSrvr] INFO -- Node 3: Data usage is 78%(284508 32K pages of total 360448) 2012-04-13 07:58:12 [MgmtSrvr] INFO -- Node 3: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:58:14 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 128 seconds (max lag: 221) 2012-04-13 07:58:18 [MgmtSrvr] WARNING -- Node 3: Node 79 missed heartbeat 4 2012-04-13 07:58:18 [MgmtSrvr] ALERT -- Node 3: Node 79 declared dead due to missed heartbeat 2012-04-13 07:58:18 [MgmtSrvr] INFO -- Node 3: Communication to Node 79 closed 2012-04-13 07:58:18 [MgmtSrvr] INFO -- Node 5: Communication to Node 79 closed 2012-04-13 07:58:18 [MgmtSrvr] INFO -- Node 4: Communication to Node 79 closed 2012-04-13 07:58:18 [MgmtSrvr] ALERT -- Node 4: Node 79 Disconnected 2012-04-13 07:58:18 [MgmtSrvr] ALERT -- Node 5: Node 79 Disconnected 2012-04-13 07:58:18 [MgmtSrvr] ALERT -- Node 3: Node 79 Disconnected 2012-04-13 07:58:19 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 3 2012-04-13 07:58:21 [MgmtSrvr] INFO -- Node 3: Node 61 Connected 2012-04-13 07:58:21 [MgmtSrvr] INFO -- Node 3: Node 61: API mysql-5.1.56 ndb-7.1.18 2012-04-13 07:58:21 [MgmtSrvr] INFO -- Node 3: Communication to Node 79 opened 2012-04-13 07:58:22 [MgmtSrvr] INFO -- Node 4: Communication to Node 79 opened 2012-04-13 07:58:22 [MgmtSrvr] INFO -- Node 5: Communication to Node 79 opened 2012-04-13 07:58:24 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 138 seconds (max lag: 221) 2012-04-13 07:58:32 [MgmtSrvr] INFO -- Node 4: Data usage is 78%(284511 32K pages of total 360448) 2012-04-13 07:58:32 [MgmtSrvr] INFO -- Node 4: Index usage is 25%(32933 8K pages of total 131200) 2012-04-13 07:58:34 [MgmtSrvr] WARNING -- Node 3: GCP Monitor: GCP_COMMIT lag 148 seconds (max lag: 221) 2012-04-13 07:58:34 [MgmtSrvr] WARNING -- Node 3: Node 4 missed heartbeat 4 2012-04-13 07:58:34 [MgmtSrvr] ALERT -- Node 3: Node 4 declared dead due to missed heartbeat 2012-04-13 07:58:34 [MgmtSrvr] INFO -- Node 3: Communication to Node 4 closed 2012-04-13 07:58:34 [MgmtSrvr] INFO -- Node 3: Communication to Node 6 closed 2012-04-13 07:58:34 [MgmtSrvr] INFO -- Node 5: Communication to Node 4 closed 2012-04-13 07:58:34 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 closed 2012-04-13 07:58:35 [MgmtSrvr] ALERT -- Node 5: Node 4 Disconnected 2012-04-13 07:58:35 [MgmtSrvr] ALERT -- Node 5: Node 6 Disconnected 2012-04-13 07:58:35 [MgmtSrvr] INFO -- Node 5: Communication to Node 4 closed 2012-04-13 07:58:35 [MgmtSrvr] INFO -- Node 5: Communication to Node 6 closed 2012-04-13 07:58:36 [MgmtSrvr] ALERT -- Node 5: Node 3 Disconnected 2012-04-13 07:58:36 [MgmtSrvr] ALERT -- Node 4: Forced node shutdown completed. Caused by error 2315: 'Node declared dead. See error log for details(Arbitration error). Temporary error, restart node'. 2012-04-13 07:58:36 [MgmtSrvr] ALERT -- Node 2: Node 3 Disconnected 2012-04-13 07:58:36 [MgmtSrvr] ALERT -- Node 3: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'. 2012-04-13 07:58:36 [MgmtSrvr] ALERT -- Node 2: Node 4 Disconnected 2012-04-13 07:58:37 [MgmtSrvr] ALERT -- Node 2: Node 5 Disconnected 2012-04-13 07:58:37 [MgmtSrvr] ALERT -- Node 5: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'. 2012-04-13 12:09:19 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:10:01 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:10:43 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:11:25 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:12:08 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:12:50 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:13:32 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:14:14 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:14:57 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:15:39 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:16:21 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:17:03 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:17:45 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:18:28 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:19:10 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:19:52 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:20:34 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:21:17 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:21:59 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:22:41 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:23:23 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:24:05 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:24:48 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:25:30 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:26:12 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:26:54 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:27:36 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:28:19 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:29:01 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).' 2012-04-13 12:29:43 [MgmtSrvr] WARNING -- Failed to allocate nodeid for API at 172.16.129.15. Returned eror: 'No free node id found for mysqld(API).'