2010-07-23 10:25:04 [ndbd] INFO     -- Angel pid: 1416 ndb pid: 1417
2010-07-23 10:25:04 [ndbd] INFO     -- NDB Cluster -- DB node 2
2010-07-23 10:25:04 [ndbd] INFO     -- mysql-5.1.30 ndb-6.3.20-GA --
2010-07-23 10:25:04 [ndbd] INFO     -- Configuration fetched at 10.20.67.215 port 1186
2010-07-23 10:25:04 [ndbd] INFO     -- Start initiated (mysql-5.1.30 ndb-6.3.20)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
2010-07-23 10:25:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:04 [ndbd] INFO     -- Watchdog: User time: 2  System time: 20
2010-07-23 10:25:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:04 [ndbd] INFO     -- Watchdog: User time: 3  System time: 30
2010-07-23 10:25:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:04 [ndbd] INFO     -- Watchdog: User time: 4  System time: 40
2010-07-23 10:25:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:04 [ndbd] INFO     -- Watchdog: User time: 4  System time: 51
2010-07-23 10:25:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:04 [ndbd] INFO     -- Watchdog: User time: 5  System time: 61
2010-07-23 10:25:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:05 [ndbd] INFO     -- Watchdog: User time: 6  System time: 71
2010-07-23 10:25:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:05 [ndbd] INFO     -- Watchdog: User time: 7  System time: 82
2010-07-23 10:25:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:05 [ndbd] INFO     -- Watchdog: User time: 8  System time: 92
2010-07-23 10:25:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:05 [ndbd] INFO     -- Watchdog: User time: 8  System time: 102
2010-07-23 10:25:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:05 [ndbd] INFO     -- Watchdog: User time: 9  System time: 112
2010-07-23 10:25:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:05 [ndbd] INFO     -- Watchdog: User time: 10  System time: 123
2010-07-23 10:25:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:05 [ndbd] INFO     -- Watchdog: User time: 11  System time: 133
2010-07-23 10:25:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:05 [ndbd] INFO     -- Watchdog: User time: 12  System time: 143
2010-07-23 10:25:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-23 10:25:05 [ndbd] INFO     -- Watchdog: User time: 12  System time: 153
2010-07-23 10:25:06 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 20000Mb initial: 20020Mb
Adding 8192Mb to ZONE_LO (1,262142)
Adding 8192Mb to ZONE_LO (262145,262142)
Adding 3636Mb to ZONE_HI (524289,116351)
WOPool::init(61, 9)
RWPool::init(22, 13)
RWPool::init(42, 18)
RWPool::init(62, 13)
RWPool::init(c2, 18)
RWPool::init(e2, 18)
WOPool::init(41, 12)
RWPool::init(82, 12)
RWPool::init(a2, 52)
WOPool::init(21, 10)
m_active_buckets.set(0)
Detected GCP stop(2)...sending kill to [SignalCounter: m_count=1 0000000000000004]
m_copyReason: 0 m_waiting: 0
c_copyGCISlave: sender{Data, Ref} 2 f60002 reason: 0 nextWord: 0
file[0] status: 2 type: 1 reqStatus: 0 file1: 2 1 0
File: /home/mysqlcluster/ndb/ndb_2_fs/D1/DBDIH/P0.sysfile 101661688
Last request: [ Request: file: 1007f3ba0 userRef: 0x00f70002 userData: 4 theFilePointer: 11845 action: write ]
theReportTo [ theSize: 512 theReadIndex: 497 theWriteIndex: 497 ]
theMemoryChannelPtr[ theSize: 512 theReadIndex: 24 theWriteIndex: 24 ]
All files: 
 0 (0x1007f2388): OPEN
 1 (0x1007f2b90): OPEN
 2 (0x1007f3398): CLOSED
 3 (0x1007f3ba0): OPEN
 4 (0x101659608): OPEN
 5 (0x101659e10): CLOSED
 6 (0x10165a618): CLOSED
 7 (0x10165ae20): OPEN
 8 (0x10165b628): CLOSED
 9 (0x10165be30): OPEN
10 (0x10165c638): CLOSED
11 (0x10165ce40): OPEN
12 (0x10165d648): OPEN
13 (0x10165de50): CLOSED
14 (0x10165e658): OPEN
15 (0x10165ee60): CLOSED
16 (0x10165f668): OPEN
17 (0x10165fe70): CLOSED
18 (0x101660678): CLOSED
19 (0x101660e80): CLOSED
20 (0x101661688): OPEN
21 (0x101661e90): OPEN
22 (0x101662698): CLOSED
23 (0x101662ea0): CLOSED
24 (0x1016636a8): CLOSED
25 (0x101663eb0): OPEN
26 (0x1016646b8): CLOSED
File: /home/mysqlcluster/ndb/ndb_2_fs/D2/DBDIH/P0.sysfile 10165ae20
Last request: [ Request: file: 1007f3ba0 userRef: 0x00f70002 userData: 4 theFilePointer: 11845 action: write ]
theReportTo [ theSize: 512 theReadIndex: 497 theWriteIndex: 497 ]
theMemoryChannelPtr[ theSize: 512 theReadIndex: 5 theWriteIndex: 5 ]
All files: 
 0 (0x1007f2388): OPEN
 1 (0x1007f2b90): OPEN
 2 (0x1007f3398): CLOSED
 3 (0x1007f3ba0): OPEN
 4 (0x101659608): OPEN
 5 (0x101659e10): CLOSED
 6 (0x10165a618): CLOSED
 7 (0x10165ae20): OPEN
 8 (0x10165b628): CLOSED
 9 (0x10165be30): OPEN
10 (0x10165c638): CLOSED
11 (0x10165ce40): OPEN
12 (0x10165d648): OPEN
13 (0x10165de50): CLOSED
14 (0x10165e658): OPEN
15 (0x10165ee60): CLOSED
16 (0x10165f668): OPEN
17 (0x10165fe70): CLOSED
18 (0x101660678): CLOSED
19 (0x101660e80): CLOSED
20 (0x101661688): OPEN
21 (0x101661e90): OPEN
22 (0x101662698): CLOSED
23 (0x101662ea0): CLOSED
24 (0x1016636a8): CLOSED
25 (0x101663eb0): OPEN
26 (0x1016646b8): CLOSED
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=1 0000000000000004]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
2010-07-27 17:02:24 [ndbd] INFO     -- Node 2 killed this node because GCP stop was detected
2010-07-27 17:02:24 [ndbd] INFO     -- NDBCNTR (Line: 249) 0x0000000e
2010-07-27 17:02:24 [ndbd] INFO     -- Error handler shutting down system
2010-07-27 17:08:51 [ndbd] INFO     -- Error handler shutdown completed - exiting
2010-07-27 17:08:56 [ndbd] ALERT    -- Node 2: Forced node shutdown completed. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2010-07-27 17:14:50 [ndbd] INFO     -- Angel pid: 14121 ndb pid: 14122
2010-07-27 17:14:50 [ndbd] INFO     -- NDB Cluster -- DB node 2
2010-07-27 17:14:50 [ndbd] INFO     -- mysql-5.1.30 ndb-6.3.20-GA --
2010-07-27 17:14:50 [ndbd] INFO     -- Configuration fetched at 10.20.67.215 port 1186
2010-07-27 17:14:50 [ndbd] INFO     -- Start initiated (mysql-5.1.30 ndb-6.3.20)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
2010-07-27 17:14:50 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:50 [ndbd] INFO     -- Watchdog: User time: 2  System time: 21
2010-07-27 17:14:50 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:50 [ndbd] INFO     -- Watchdog: User time: 3  System time: 31
2010-07-27 17:14:50 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:50 [ndbd] INFO     -- Watchdog: User time: 4  System time: 41
2010-07-27 17:14:50 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:50 [ndbd] INFO     -- Watchdog: User time: 5  System time: 51
2010-07-27 17:14:50 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:50 [ndbd] INFO     -- Watchdog: User time: 5  System time: 62
2010-07-27 17:14:51 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:51 [ndbd] INFO     -- Watchdog: User time: 6  System time: 72
2010-07-27 17:14:51 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:51 [ndbd] INFO     -- Watchdog: User time: 7  System time: 82
2010-07-27 17:14:51 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:51 [ndbd] INFO     -- Watchdog: User time: 8  System time: 92
2010-07-27 17:14:51 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:51 [ndbd] INFO     -- Watchdog: User time: 9  System time: 103
2010-07-27 17:14:51 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:51 [ndbd] INFO     -- Watchdog: User time: 9  System time: 113
2010-07-27 17:14:51 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:51 [ndbd] INFO     -- Watchdog: User time: 10  System time: 123
2010-07-27 17:14:51 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:51 [ndbd] INFO     -- Watchdog: User time: 11  System time: 134
2010-07-27 17:14:51 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:51 [ndbd] INFO     -- Watchdog: User time: 12  System time: 144
2010-07-27 17:14:51 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:14:51 [ndbd] INFO     -- Watchdog: User time: 13  System time: 154
2010-07-27 17:14:51 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 20000Mb initial: 20020Mb
Adding 8192Mb to ZONE_LO (1,262142)
Adding 8192Mb to ZONE_LO (262145,262142)
Adding 3636Mb to ZONE_HI (524289,116351)
WOPool::init(61, 9)
RWPool::init(22, 13)
RWPool::init(42, 18)
RWPool::init(62, 13)
RWPool::init(c2, 18)
RWPool::init(e2, 18)
WOPool::init(41, 12)
RWPool::init(82, 12)
RWPool::init(a2, 52)
WOPool::init(21, 10)
RESTORE table: 0 2050 rows applied
RESTORE table: 1 15 rows applied
RESTORE table: 2 12 rows applied
RESTORE table: 3 9 rows applied
RESTORE table: 4 1 rows applied
RESTORE table: 5 1 rows applied
RESTORE table: 7 10009211 rows applied
RESTORE table: 10 71 rows applied
RESTORE table: 14 0 rows applied
RESTORE table: 21 0 rows applied
RESTORE table: 27 0 rows applied
RESTORE table: 33 0 rows applied
RESTORE table: 39 0 rows applied
2010-07-27 17:39:04 [ndbd] ALERT    -- Node 2: Forced node shutdown completed. Initiated by signal 9.
2010-07-27 17:40:02 [ndbd] INFO     -- Angel pid: 15709 ndb pid: 15710
2010-07-27 17:40:02 [ndbd] INFO     -- NDB Cluster -- DB node 2
2010-07-27 17:40:02 [ndbd] INFO     -- mysql-5.1.30 ndb-6.3.20-GA --
2010-07-27 17:40:02 [ndbd] INFO     -- Configuration fetched at 10.20.67.215 port 1186
2010-07-27 17:40:02 [ndbd] INFO     -- Start initiated (mysql-5.1.30 ndb-6.3.20)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
2010-07-27 17:40:02 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:02 [ndbd] INFO     -- Watchdog: User time: 2  System time: 21
2010-07-27 17:40:02 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:02 [ndbd] INFO     -- Watchdog: User time: 3  System time: 31
2010-07-27 17:40:03 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:03 [ndbd] INFO     -- Watchdog: User time: 4  System time: 41
2010-07-27 17:40:03 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:03 [ndbd] INFO     -- Watchdog: User time: 4  System time: 51
2010-07-27 17:40:03 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:03 [ndbd] INFO     -- Watchdog: User time: 5  System time: 62
2010-07-27 17:40:03 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:03 [ndbd] INFO     -- Watchdog: User time: 6  System time: 72
2010-07-27 17:40:03 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:03 [ndbd] INFO     -- Watchdog: User time: 7  System time: 82
2010-07-27 17:40:03 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:03 [ndbd] INFO     -- Watchdog: User time: 8  System time: 92
2010-07-27 17:40:03 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:03 [ndbd] INFO     -- Watchdog: User time: 8  System time: 103
2010-07-27 17:40:03 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:03 [ndbd] INFO     -- Watchdog: User time: 9  System time: 113
2010-07-27 17:40:03 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:03 [ndbd] INFO     -- Watchdog: User time: 10  System time: 123
2010-07-27 17:40:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:04 [ndbd] INFO     -- Watchdog: User time: 11  System time: 134
2010-07-27 17:40:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:04 [ndbd] INFO     -- Watchdog: User time: 12  System time: 144
2010-07-27 17:40:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:04 [ndbd] INFO     -- Watchdog: User time: 13  System time: 154
2010-07-27 17:40:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:04 [ndbd] INFO     -- Watchdog: User time: 13  System time: 164
2010-07-27 17:40:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:04 [ndbd] INFO     -- Watchdog: User time: 14  System time: 175
2010-07-27 17:40:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:04 [ndbd] INFO     -- Watchdog: User time: 15  System time: 185
2010-07-27 17:40:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:04 [ndbd] INFO     -- Watchdog: User time: 16  System time: 195
2010-07-27 17:40:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:04 [ndbd] INFO     -- Watchdog: User time: 17  System time: 205
2010-07-27 17:40:04 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:04 [ndbd] INFO     -- Watchdog: User time: 17  System time: 216
2010-07-27 17:40:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:05 [ndbd] INFO     -- Watchdog: User time: 18  System time: 226
2010-07-27 17:40:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:05 [ndbd] INFO     -- Watchdog: User time: 19  System time: 236
2010-07-27 17:40:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:05 [ndbd] INFO     -- Watchdog: User time: 21  System time: 240
2010-07-27 17:40:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:05 [ndbd] INFO     -- Watchdog: User time: 22  System time: 258
2010-07-27 17:40:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:05 [ndbd] INFO     -- Watchdog: User time: 23  System time: 269
2010-07-27 17:40:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:05 [ndbd] INFO     -- Watchdog: User time: 24  System time: 279
2010-07-27 17:40:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:05 [ndbd] INFO     -- Watchdog: User time: 25  System time: 289
2010-07-27 17:40:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:05 [ndbd] INFO     -- Watchdog: User time: 25  System time: 299
2010-07-27 17:40:05 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:05 [ndbd] INFO     -- Watchdog: User time: 26  System time: 310
2010-07-27 17:40:06 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:06 [ndbd] INFO     -- Watchdog: User time: 27  System time: 320
2010-07-27 17:40:06 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-27 17:40:06 [ndbd] INFO     -- Watchdog: User time: 28  System time: 330
2010-07-27 17:40:06 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 20000Mb initial: 20020Mb
Adding 8192Mb to ZONE_LO (1,262142)
Adding 8192Mb to ZONE_LO (262145,262142)
Adding 3636Mb to ZONE_HI (524289,116351)
WOPool::init(61, 9)
RWPool::init(22, 13)
RWPool::init(42, 18)
RWPool::init(62, 13)
RWPool::init(c2, 18)
RWPool::init(e2, 18)
WOPool::init(41, 12)
RWPool::init(82, 12)
RWPool::init(a2, 52)
WOPool::init(21, 10)
RESTORE table: 0 2050 rows applied
RESTORE table: 1 15 rows applied
RESTORE table: 2 12 rows applied
RESTORE table: 3 9 rows applied
RESTORE table: 4 1 rows applied
RESTORE table: 5 1 rows applied
RESTORE table: 7 10009211 rows applied
RESTORE table: 10 71 rows applied
RESTORE table: 14 0 rows applied
RESTORE table: 21 0 rows applied
RESTORE table: 27 0 rows applied
RESTORE table: 33 0 rows applied
RESTORE table: 39 0 rows applied
RESTORE table: 45 40515297 rows applied
RESTORE table: 51 0 rows applied
RESTORE table: 53 555 rows applied
RESTORE table: 56 0 rows applied
m_active_buckets.set(0)
Detected GCP stop(2)...sending kill to [SignalCounter: m_count=1 0000000000000004]
m_copyReason: 0 m_waiting: 0
c_copyGCISlave: sender{Data, Ref} 2 f60002 reason: 0 nextWord: 0
file[0] status: 2 type: 1 reqStatus: 0 file1: 2 1 0
File: /home/mysqlcluster/ndb/ndb_2_fs/D1/DBDIH/P0.sysfile 10165be30
Last request: [ Request: file: 10165b628 userRef: 0x00f70002 userData: 217 theFilePointer: 407 action: close ]
theReportTo [ theSize: 512 theReadIndex: 80 theWriteIndex: 80 ]
theMemoryChannelPtr[ theSize: 512 theReadIndex: 46 theWriteIndex: 46 ]
All files: 
 0 (0x1007f2388): CLOSED
 1 (0x1007f2b90): CLOSED
 2 (0x1007f3398): OPEN
 3 (0x1007f3ba0): OPEN
 4 (0x101659608): OPEN
 5 (0x101659e10): OPEN
 6 (0x10165a618): OPEN
 7 (0x10165ae20): OPEN
 8 (0x10165b628): CLOSED
 9 (0x10165be30): OPEN
10 (0x10165c638): CLOSED
11 (0x10165ce40): CLOSED
12 (0x10165d648): CLOSED
13 (0x10165de50): OPEN
14 (0x10165e658): OPEN
15 (0x10165ee60): OPEN
16 (0x10165f668): OPEN
17 (0x10165fe70): OPEN
18 (0x101660678): CLOSED
19 (0x101660e80): CLOSED
20 (0x101661688): CLOSED
21 (0x101661e90): CLOSED
22 (0x101662698): OPEN
23 (0x101662ea0): CLOSED
24 (0x1016636a8): CLOSED
25 (0x101663eb0): CLOSED
26 (0x1016646b8): CLOSED
File: /home/mysqlcluster/ndb/ndb_2_fs/D2/DBDIH/P0.sysfile 10165de50
Last request: [ Request: file: 10165b628 userRef: 0x00f70002 userData: 217 theFilePointer: 407 action: close ]
theReportTo [ theSize: 512 theReadIndex: 80 theWriteIndex: 80 ]
theMemoryChannelPtr[ theSize: 512 theReadIndex: 46 theWriteIndex: 46 ]
All files: 
 0 (0x1007f2388): CLOSED
 1 (0x1007f2b90): CLOSED
 2 (0x1007f3398): OPEN
 3 (0x1007f3ba0): OPEN
 4 (0x101659608): OPEN
 5 (0x101659e10): OPEN
 6 (0x10165a618): OPEN
 7 (0x10165ae20): OPEN
 8 (0x10165b628): CLOSED
 9 (0x10165be30): OPEN
10 (0x10165c638): CLOSED
11 (0x10165ce40): CLOSED
12 (0x10165d648): CLOSED
13 (0x10165de50): OPEN
14 (0x10165e658): OPEN
15 (0x10165ee60): OPEN
16 (0x10165f668): OPEN
17 (0x10165fe70): OPEN
18 (0x101660678): CLOSED
19 (0x101660e80): CLOSED
20 (0x101661688): CLOSED
21 (0x101661e90): CLOSED
22 (0x101662698): OPEN
23 (0x101662ea0): CLOSED
24 (0x1016636a8): CLOSED
25 (0x101663eb0): CLOSED
26 (0x1016646b8): CLOSED
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=1 0000000000000004]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
2010-07-28 01:02:20 [ndbd] INFO     -- Node 2 killed this node because GCP stop was detected
2010-07-28 01:02:20 [ndbd] INFO     -- NDBCNTR (Line: 249) 0x0000000a
2010-07-28 01:02:20 [ndbd] INFO     -- Error handler shutting down system
2010-07-28 01:10:36 [ndbd] INFO     -- Error handler shutdown completed - exiting
2010-07-28 01:10:42 [ndbd] ALERT    -- Node 2: Forced node shutdown completed. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2010-07-28 02:01:39 [ndbd] INFO     -- Angel pid: 22079 ndb pid: 22080
2010-07-28 02:01:39 [ndbd] INFO     -- NDB Cluster -- DB node 2
2010-07-28 02:01:39 [ndbd] INFO     -- mysql-5.1.30 ndb-6.3.20-GA --
2010-07-28 02:01:39 [ndbd] INFO     -- Configuration fetched at 10.20.67.215 port 1186
2010-07-28 02:01:39 [ndbd] INFO     -- Start initiated (mysql-5.1.30 ndb-6.3.20)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
2010-07-28 02:01:39 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:39 [ndbd] INFO     -- Watchdog: User time: 2  System time: 20
2010-07-28 02:01:39 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:39 [ndbd] INFO     -- Watchdog: User time: 3  System time: 31
2010-07-28 02:01:39 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:39 [ndbd] INFO     -- Watchdog: User time: 4  System time: 41
2010-07-28 02:01:39 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:39 [ndbd] INFO     -- Watchdog: User time: 4  System time: 51
2010-07-28 02:01:39 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:39 [ndbd] INFO     -- Watchdog: User time: 5  System time: 61
2010-07-28 02:01:40 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:40 [ndbd] INFO     -- Watchdog: User time: 6  System time: 72
2010-07-28 02:01:40 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:40 [ndbd] INFO     -- Watchdog: User time: 7  System time: 82
2010-07-28 02:01:40 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:40 [ndbd] INFO     -- Watchdog: User time: 8  System time: 92
2010-07-28 02:01:40 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:40 [ndbd] INFO     -- Watchdog: User time: 9  System time: 102
2010-07-28 02:01:40 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:40 [ndbd] INFO     -- Watchdog: User time: 9  System time: 113
2010-07-28 02:01:40 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:40 [ndbd] INFO     -- Watchdog: User time: 10  System time: 123
2010-07-28 02:01:40 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:40 [ndbd] INFO     -- Watchdog: User time: 11  System time: 133
2010-07-28 02:01:40 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:40 [ndbd] INFO     -- Watchdog: User time: 12  System time: 144
2010-07-28 02:01:40 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:40 [ndbd] INFO     -- Watchdog: User time: 13  System time: 154
2010-07-28 02:01:41 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:41 [ndbd] INFO     -- Watchdog: User time: 13  System time: 164
2010-07-28 02:01:41 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:41 [ndbd] INFO     -- Watchdog: User time: 14  System time: 174
2010-07-28 02:01:41 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:41 [ndbd] INFO     -- Watchdog: User time: 15  System time: 185
2010-07-28 02:01:41 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:41 [ndbd] INFO     -- Watchdog: User time: 16  System time: 195
2010-07-28 02:01:41 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:41 [ndbd] INFO     -- Watchdog: User time: 17  System time: 205
2010-07-28 02:01:41 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:41 [ndbd] INFO     -- Watchdog: User time: 17  System time: 215
2010-07-28 02:01:41 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:41 [ndbd] INFO     -- Watchdog: User time: 18  System time: 226
2010-07-28 02:01:41 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:41 [ndbd] INFO     -- Watchdog: User time: 19  System time: 235
2010-07-28 02:01:41 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:41 [ndbd] INFO     -- Watchdog: User time: 19  System time: 247
2010-07-28 02:01:42 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:42 [ndbd] INFO     -- Watchdog: User time: 20  System time: 258
2010-07-28 02:01:42 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:42 [ndbd] INFO     -- Watchdog: User time: 21  System time: 268
2010-07-28 02:01:42 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:42 [ndbd] INFO     -- Watchdog: User time: 22  System time: 279
2010-07-28 02:01:42 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:42 [ndbd] INFO     -- Watchdog: User time: 23  System time: 289
2010-07-28 02:01:42 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:42 [ndbd] INFO     -- Watchdog: User time: 23  System time: 299
2010-07-28 02:01:42 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:42 [ndbd] INFO     -- Watchdog: User time: 24  System time: 309
2010-07-28 02:01:42 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:01:42 [ndbd] INFO     -- Watchdog: User time: 25  System time: 320
2010-07-28 02:01:42 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 20000Mb initial: 20020Mb
Adding 8192Mb to ZONE_LO (1,262142)
Adding 8192Mb to ZONE_LO (262145,262142)
Adding 3636Mb to ZONE_HI (524289,116351)
WOPool::init(61, 9)
RWPool::init(22, 13)
RWPool::init(42, 18)
RWPool::init(62, 13)
RWPool::init(c2, 18)
RWPool::init(e2, 18)
WOPool::init(41, 12)
RWPool::init(82, 12)
RWPool::init(a2, 52)
WOPool::init(21, 10)
RESTORE table: 0 2050 rows applied
RESTORE table: 1 15 rows applied
RESTORE table: 2 12 rows applied
RESTORE table: 3 9 rows applied
RESTORE table: 4 1 rows applied
RESTORE table: 5 1 rows applied
2010-07-28 02:05:03 [ndbd] INFO     -- Shutdown initiated
2010-07-28 02:05:03 [ndbd] INFO     -- Shutdown completed - exiting
2010-07-28 02:05:05 [ndbd] INFO     -- Angel shutting down
2010-07-28 02:05:05 [ndbd] INFO     -- Node 2: Node shutdown completed.
2010-07-28 02:05:17 [ndbd] INFO     -- Angel pid: 22333 ndb pid: 22334
2010-07-28 02:05:17 [ndbd] INFO     -- NDB Cluster -- DB node 2
2010-07-28 02:05:17 [ndbd] INFO     -- mysql-5.1.30 ndb-6.3.20-GA --
2010-07-28 02:05:17 [ndbd] INFO     -- Configuration fetched at 10.20.67.215 port 1186
2010-07-28 02:05:17 [ndbd] INFO     -- Start initiated (mysql-5.1.30 ndb-6.3.20)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
2010-07-28 02:05:18 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:18 [ndbd] INFO     -- Watchdog: User time: 2  System time: 21
2010-07-28 02:05:18 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:18 [ndbd] INFO     -- Watchdog: User time: 3  System time: 31
2010-07-28 02:05:18 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:18 [ndbd] INFO     -- Watchdog: User time: 4  System time: 41
2010-07-28 02:05:18 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:18 [ndbd] INFO     -- Watchdog: User time: 4  System time: 52
2010-07-28 02:05:18 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:18 [ndbd] INFO     -- Watchdog: User time: 5  System time: 62
2010-07-28 02:05:18 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:18 [ndbd] INFO     -- Watchdog: User time: 6  System time: 72
2010-07-28 02:05:18 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:18 [ndbd] INFO     -- Watchdog: User time: 7  System time: 82
2010-07-28 02:05:18 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:18 [ndbd] INFO     -- Watchdog: User time: 8  System time: 93
2010-07-28 02:05:19 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:19 [ndbd] INFO     -- Watchdog: User time: 9  System time: 103
2010-07-28 02:05:19 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:19 [ndbd] INFO     -- Watchdog: User time: 9  System time: 113
2010-07-28 02:05:19 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:19 [ndbd] INFO     -- Watchdog: User time: 10  System time: 124
2010-07-28 02:05:19 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:19 [ndbd] INFO     -- Watchdog: User time: 11  System time: 134
2010-07-28 02:05:19 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:19 [ndbd] INFO     -- Watchdog: User time: 12  System time: 144
2010-07-28 02:05:19 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:19 [ndbd] INFO     -- Watchdog: User time: 13  System time: 154
2010-07-28 02:05:19 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:19 [ndbd] INFO     -- Watchdog: User time: 13  System time: 165
2010-07-28 02:05:19 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:19 [ndbd] INFO     -- Watchdog: User time: 14  System time: 175
2010-07-28 02:05:19 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:19 [ndbd] INFO     -- Watchdog: User time: 15  System time: 185
2010-07-28 02:05:20 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:20 [ndbd] INFO     -- Watchdog: User time: 16  System time: 195
2010-07-28 02:05:20 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:20 [ndbd] INFO     -- Watchdog: User time: 17  System time: 206
2010-07-28 02:05:20 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:20 [ndbd] INFO     -- Watchdog: User time: 17  System time: 216
2010-07-28 02:05:20 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:20 [ndbd] INFO     -- Watchdog: User time: 18  System time: 226
2010-07-28 02:05:20 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:20 [ndbd] INFO     -- Watchdog: User time: 19  System time: 235
2010-07-28 02:05:20 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:20 [ndbd] INFO     -- Watchdog: User time: 19  System time: 247
2010-07-28 02:05:20 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:20 [ndbd] INFO     -- Watchdog: User time: 20  System time: 258
2010-07-28 02:05:20 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:20 [ndbd] INFO     -- Watchdog: User time: 21  System time: 268
2010-07-28 02:05:20 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:20 [ndbd] INFO     -- Watchdog: User time: 22  System time: 278
2010-07-28 02:05:21 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:21 [ndbd] INFO     -- Watchdog: User time: 23  System time: 288
2010-07-28 02:05:21 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:21 [ndbd] INFO     -- Watchdog: User time: 23  System time: 299
2010-07-28 02:05:21 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:21 [ndbd] INFO     -- Watchdog: User time: 24  System time: 310
2010-07-28 02:05:21 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:21 [ndbd] INFO     -- Watchdog: User time: 25  System time: 320
2010-07-28 02:05:21 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:21 [ndbd] INFO     -- Watchdog: User time: 26  System time: 330
2010-07-28 02:05:21 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:21 [ndbd] INFO     -- Watchdog: User time: 27  System time: 341
2010-07-28 02:05:21 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:21 [ndbd] INFO     -- Watchdog: User time: 27  System time: 351
2010-07-28 02:05:21 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:21 [ndbd] INFO     -- Watchdog: User time: 28  System time: 361
2010-07-28 02:05:21 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:21 [ndbd] INFO     -- Watchdog: User time: 29  System time: 371
2010-07-28 02:05:22 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:22 [ndbd] INFO     -- Watchdog: User time: 30  System time: 382
2010-07-28 02:05:22 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:22 [ndbd] INFO     -- Watchdog: User time: 31  System time: 392
2010-07-28 02:05:22 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:22 [ndbd] INFO     -- Watchdog: User time: 32  System time: 402
2010-07-28 02:05:22 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:22 [ndbd] INFO     -- Watchdog: User time: 32  System time: 412
2010-07-28 02:05:22 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:22 [ndbd] INFO     -- Watchdog: User time: 33  System time: 423
2010-07-28 02:05:22 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:22 [ndbd] INFO     -- Watchdog: User time: 34  System time: 433
2010-07-28 02:05:22 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:22 [ndbd] INFO     -- Watchdog: User time: 35  System time: 443
2010-07-28 02:05:22 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:22 [ndbd] INFO     -- Watchdog: User time: 36  System time: 454
2010-07-28 02:05:22 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:22 [ndbd] INFO     -- Watchdog: User time: 36  System time: 464
2010-07-28 02:05:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:23 [ndbd] INFO     -- Watchdog: User time: 37  System time: 474
2010-07-28 02:05:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:23 [ndbd] INFO     -- Watchdog: User time: 38  System time: 487
2010-07-28 02:05:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:23 [ndbd] INFO     -- Watchdog: User time: 39  System time: 497
2010-07-28 02:05:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:23 [ndbd] INFO     -- Watchdog: User time: 39  System time: 507
2010-07-28 02:05:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:23 [ndbd] INFO     -- Watchdog: User time: 40  System time: 518
2010-07-28 02:05:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:23 [ndbd] INFO     -- Watchdog: User time: 41  System time: 528
2010-07-28 02:05:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:23 [ndbd] INFO     -- Watchdog: User time: 42  System time: 538
2010-07-28 02:05:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:23 [ndbd] INFO     -- Watchdog: User time: 43  System time: 548
2010-07-28 02:05:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:23 [ndbd] INFO     -- Watchdog: User time: 43  System time: 559
2010-07-28 02:05:24 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:24 [ndbd] INFO     -- Watchdog: User time: 44  System time: 569
2010-07-28 02:05:24 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:24 [ndbd] INFO     -- Watchdog: User time: 45  System time: 579
2010-07-28 02:05:24 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:24 [ndbd] INFO     -- Watchdog: User time: 46  System time: 590
2010-07-28 02:05:24 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:24 [ndbd] INFO     -- Watchdog: User time: 47  System time: 600
2010-07-28 02:05:24 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:24 [ndbd] INFO     -- Watchdog: User time: 47  System time: 610
2010-07-28 02:05:24 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:24 [ndbd] INFO     -- Watchdog: User time: 48  System time: 620
2010-07-28 02:05:24 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:24 [ndbd] INFO     -- Watchdog: User time: 49  System time: 631
2010-07-28 02:05:24 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:24 [ndbd] INFO     -- Watchdog: User time: 50  System time: 641
2010-07-28 02:05:24 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 20000Mb initial: 20020Mb
Adding 8192Mb to ZONE_LO (1,262142)
Adding 8192Mb to ZONE_LO (262145,262142)
Adding 3636Mb to ZONE_HI (524289,116351)
WOPool::init(61, 9)
RWPool::init(22, 13)
RWPool::init(42, 18)
RWPool::init(62, 13)
RWPool::init(c2, 18)
RWPool::init(e2, 18)
WOPool::init(41, 12)
RWPool::init(82, 12)
RWPool::init(a2, 52)
2010-07-28 02:05:28 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 02:05:28 [ndbd] INFO     -- Watchdog: User time: 220  System time: 802
WOPool::init(21, 10)
RESTORE table: 0 2050 rows applied
RESTORE table: 1 15 rows applied
RESTORE table: 2 12 rows applied
RESTORE table: 3 9 rows applied
RESTORE table: 4 1 rows applied
RESTORE table: 5 1 rows applied
RESTORE table: 7 10009211 rows applied
RESTORE table: 10 74 rows applied
RESTORE table: 14 0 rows applied
RESTORE table: 21 0 rows applied
RESTORE table: 27 0 rows applied
RESTORE table: 33 0 rows applied
RESTORE table: 39 0 rows applied
RESTORE table: 45 40615299 rows applied
RESTORE table: 51 0 rows applied
RESTORE table: 53 578 rows applied
RESTORE table: 56 0 rows applied
m_active_buckets.set(0)
2010-07-28 03:50:50 [ndbd] INFO     -- Shutdown initiated
2010-07-28 03:50:50 [ndbd] INFO     -- Shutdown completed - exiting
2010-07-28 03:50:55 [ndbd] INFO     -- Angel shutting down
2010-07-28 03:50:55 [ndbd] INFO     -- Node 2: Node shutdown completed.
2010-07-28 03:54:10 [ndbd] INFO     -- Angel pid: 29008 ndb pid: 29009
2010-07-28 03:54:10 [ndbd] INFO     -- NDB Cluster -- DB node 2
2010-07-28 03:54:10 [ndbd] INFO     -- mysql-5.1.30 ndb-6.3.20-GA --
2010-07-28 03:54:10 [ndbd] INFO     -- Configuration fetched at 10.20.67.215 port 1186
2010-07-28 03:54:10 [ndbd] INFO     -- Start initiated (mysql-5.1.30 ndb-6.3.20)
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer
2010-07-28 03:54:10 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:10 [ndbd] INFO     -- Watchdog: User time: 2  System time: 20
2010-07-28 03:54:10 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:10 [ndbd] INFO     -- Watchdog: User time: 3  System time: 30
2010-07-28 03:54:10 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:10 [ndbd] INFO     -- Watchdog: User time: 4  System time: 41
2010-07-28 03:54:10 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:10 [ndbd] INFO     -- Watchdog: User time: 4  System time: 51
2010-07-28 03:54:10 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:10 [ndbd] INFO     -- Watchdog: User time: 5  System time: 61
2010-07-28 03:54:11 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:11 [ndbd] INFO     -- Watchdog: User time: 6  System time: 72
2010-07-28 03:54:11 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:11 [ndbd] INFO     -- Watchdog: User time: 7  System time: 82
2010-07-28 03:54:11 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:11 [ndbd] INFO     -- Watchdog: User time: 8  System time: 92
2010-07-28 03:54:11 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:11 [ndbd] INFO     -- Watchdog: User time: 8  System time: 102
2010-07-28 03:54:11 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:11 [ndbd] INFO     -- Watchdog: User time: 9  System time: 113
2010-07-28 03:54:11 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:11 [ndbd] INFO     -- Watchdog: User time: 10  System time: 123
2010-07-28 03:54:11 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:11 [ndbd] INFO     -- Watchdog: User time: 11  System time: 133
2010-07-28 03:54:11 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:11 [ndbd] INFO     -- Watchdog: User time: 12  System time: 143
2010-07-28 03:54:11 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:11 [ndbd] INFO     -- Watchdog: User time: 12  System time: 154
2010-07-28 03:54:12 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:12 [ndbd] INFO     -- Watchdog: User time: 13  System time: 164
2010-07-28 03:54:12 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:12 [ndbd] INFO     -- Watchdog: User time: 14  System time: 174
2010-07-28 03:54:12 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:12 [ndbd] INFO     -- Watchdog: User time: 15  System time: 185
2010-07-28 03:54:12 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:12 [ndbd] INFO     -- Watchdog: User time: 16  System time: 195
2010-07-28 03:54:12 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:12 [ndbd] INFO     -- Watchdog: User time: 16  System time: 205
2010-07-28 03:54:12 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:12 [ndbd] INFO     -- Watchdog: User time: 17  System time: 215
2010-07-28 03:54:12 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:12 [ndbd] INFO     -- Watchdog: User time: 18  System time: 226
2010-07-28 03:54:12 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:12 [ndbd] INFO     -- Watchdog: User time: 19  System time: 236
2010-07-28 03:54:12 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:12 [ndbd] INFO     -- Watchdog: User time: 19  System time: 247
2010-07-28 03:54:13 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:13 [ndbd] INFO     -- Watchdog: User time: 20  System time: 257
2010-07-28 03:54:13 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:13 [ndbd] INFO     -- Watchdog: User time: 21  System time: 267
2010-07-28 03:54:13 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:13 [ndbd] INFO     -- Watchdog: User time: 22  System time: 278
2010-07-28 03:54:13 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:13 [ndbd] INFO     -- Watchdog: User time: 22  System time: 288
2010-07-28 03:54:13 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:13 [ndbd] INFO     -- Watchdog: User time: 23  System time: 298
2010-07-28 03:54:13 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:13 [ndbd] INFO     -- Watchdog: User time: 24  System time: 308
2010-07-28 03:54:13 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:13 [ndbd] INFO     -- Watchdog: User time: 25  System time: 319
2010-07-28 03:54:13 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:13 [ndbd] INFO     -- Watchdog: User time: 26  System time: 329
2010-07-28 03:54:13 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:13 [ndbd] INFO     -- Watchdog: User time: 26  System time: 339
2010-07-28 03:54:14 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:14 [ndbd] INFO     -- Watchdog: User time: 27  System time: 349
2010-07-28 03:54:14 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:14 [ndbd] INFO     -- Watchdog: User time: 28  System time: 360
2010-07-28 03:54:14 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:14 [ndbd] INFO     -- Watchdog: User time: 29  System time: 370
2010-07-28 03:54:14 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:14 [ndbd] INFO     -- Watchdog: User time: 30  System time: 380
2010-07-28 03:54:14 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:14 [ndbd] INFO     -- Watchdog: User time: 30  System time: 390
2010-07-28 03:54:14 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:14 [ndbd] INFO     -- Watchdog: User time: 31  System time: 401
2010-07-28 03:54:14 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:14 [ndbd] INFO     -- Watchdog: User time: 32  System time: 411
2010-07-28 03:54:14 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:14 [ndbd] INFO     -- Watchdog: User time: 33  System time: 421
2010-07-28 03:54:14 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:14 [ndbd] INFO     -- Watchdog: User time: 34  System time: 431
2010-07-28 03:54:15 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:15 [ndbd] INFO     -- Watchdog: User time: 35  System time: 442
2010-07-28 03:54:15 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:15 [ndbd] INFO     -- Watchdog: User time: 35  System time: 452
2010-07-28 03:54:15 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:15 [ndbd] INFO     -- Watchdog: User time: 36  System time: 462
2010-07-28 03:54:15 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:15 [ndbd] INFO     -- Watchdog: User time: 37  System time: 473
2010-07-28 03:54:15 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:15 [ndbd] INFO     -- Watchdog: User time: 38  System time: 483
2010-07-28 03:54:15 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:15 [ndbd] INFO     -- Watchdog: User time: 38  System time: 493
2010-07-28 03:54:15 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:15 [ndbd] INFO     -- Watchdog: User time: 39  System time: 504
2010-07-28 03:54:15 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:15 [ndbd] INFO     -- Watchdog: User time: 40  System time: 514
2010-07-28 03:54:15 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:15 [ndbd] INFO     -- Watchdog: User time: 41  System time: 524
2010-07-28 03:54:16 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:16 [ndbd] INFO     -- Watchdog: User time: 42  System time: 536
2010-07-28 03:54:16 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:16 [ndbd] INFO     -- Watchdog: User time: 42  System time: 546
2010-07-28 03:54:16 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:16 [ndbd] INFO     -- Watchdog: User time: 43  System time: 556
2010-07-28 03:54:16 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:16 [ndbd] INFO     -- Watchdog: User time: 44  System time: 566
2010-07-28 03:54:16 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:16 [ndbd] INFO     -- Watchdog: User time: 45  System time: 576
2010-07-28 03:54:16 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:16 [ndbd] INFO     -- Watchdog: User time: 46  System time: 587
2010-07-28 03:54:16 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:16 [ndbd] INFO     -- Watchdog: User time: 46  System time: 597
2010-07-28 03:54:16 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:16 [ndbd] INFO     -- Watchdog: User time: 47  System time: 607
2010-07-28 03:54:16 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:16 [ndbd] INFO     -- Watchdog: User time: 48  System time: 617
2010-07-28 03:54:17 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:17 [ndbd] INFO     -- Watchdog: User time: 49  System time: 628
2010-07-28 03:54:17 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:17 [ndbd] INFO     -- Watchdog: User time: 50  System time: 638
2010-07-28 03:54:17 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 20000Mb initial: 20020Mb
Adding 8192Mb to ZONE_LO (1,262142)
Adding 8192Mb to ZONE_LO (262145,262142)
Adding 3636Mb to ZONE_HI (524289,116351)
WOPool::init(61, 9)
RWPool::init(22, 13)
2010-07-28 03:54:23 [ndbd] WARNING  -- Ndb kernel is stuck in: Allocating memory
2010-07-28 03:54:23 [ndbd] INFO     -- Watchdog: User time: 210  System time: 1157
RWPool::init(42, 18)
RWPool::init(62, 13)
RWPool::init(c2, 18)
RWPool::init(e2, 18)
WOPool::init(41, 12)
RWPool::init(82, 12)
RWPool::init(a2, 52)
2010-07-28 03:54:29 [ndbd] WARNING  -- Ndb kernel is stuck in: Job Handling
2010-07-28 03:54:29 [ndbd] INFO     -- Watchdog: User time: 411  System time: 1497
WOPool::init(21, 10)
RESTORE table: 0 2050 rows applied
RESTORE table: 1 15 rows applied
RESTORE table: 2 12 rows applied
RESTORE table: 3 9 rows applied
RESTORE table: 4 1 rows applied
RESTORE table: 5 1 rows applied
RESTORE table: 7 10009211 rows applied
RESTORE table: 10 74 rows applied
RESTORE table: 14 0 rows applied
RESTORE table: 21 0 rows applied
RESTORE table: 27 0 rows applied
RESTORE table: 33 0 rows applied
RESTORE table: 39 0 rows applied
RESTORE table: 45 40615299 rows applied
RESTORE table: 51 0 rows applied
RESTORE table: 53 578 rows applied
RESTORE table: 56 0 rows applied
m_active_buckets.set(0)
2010-07-28 10:44:28 [ndbd] INFO     -- Shutdown initiated
2010-07-28 10:44:28 [ndbd] INFO     -- Shutdown completed - exiting
2010-07-28 10:44:38 [ndbd] INFO     -- Angel shutting down
2010-07-28 10:44:38 [ndbd] INFO     -- Node 2: Node shutdown completed.