node2 out log:
==============
2009-10-15 02:31:06 [ndbd] INFO     -- Watchdog: User time: 818727  System time: 571448
2009-10-15 02:31:06 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Performing Send elapsed=1000
2009-10-15 02:31:06 [ndbd] INFO     -- Watchdog: User time: 818727  System time: 571458
WARNING: timerHandlingLab now: 7154809235 sent: 7154808086 diff: 1149
2009-10-15 02:31:07 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=100
2009-10-15 02:31:07 [ndbd] INFO     -- Watchdog: User time: 818841  System time: 571468
2009-10-15 02:31:07 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=200
2009-10-15 02:31:07 [ndbd] INFO     -- Watchdog: User time: 818841  System time: 571468
2009-10-15 02:31:07 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=300
2009-10-15 02:31:07 [ndbd] INFO     -- Watchdog: User time: 818841  System time: 571468
2009-10-15 02:31:07 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=400
2009-10-15 02:31:07 [ndbd] INFO     -- Watchdog: User time: 818841  System time: 571468
2009-10-15 02:31:08 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=500
2009-10-15 02:31:08 [ndbd] INFO     -- Watchdog: User time: 818841  System time: 571468
2009-10-15 02:31:08 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=600
2009-10-15 02:31:08 [ndbd] INFO     -- Watchdog: User time: 818841  System time: 571468
2009-10-15 02:31:08 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=700
2009-10-15 02:31:08 [ndbd] INFO     -- Watchdog: User time: 818841  System time: 571468
2009-10-15 02:31:08 [ndbd] INFO     -- SimulatedBlock.cpp
2009-10-15 02:31:08 [ndbd] INFO     -- DBTC (Line: 510) 0x0000000e
2009-10-15 02:31:08 [ndbd] INFO     -- Error handler shutting down system
2009-10-15 02:31:08 [ndbd] INFO     -- Error handler shutdown completed - exiting
2009-10-15 02:31:13 [ndbd] ALERT    -- Node 2: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

###################################################################################################
node3 out log:
==============

m_gcp_save.m_counter: 1220 m_gcp_save.m_max_lag: 1220
m_micro_gcp.m_counter: 1158 m_micro_gcp.m_max_lag: 1241
m_gcp_save.m_state: 0
m_gcp_save.m_master.m_state: 0
m_micro_gcp.m_state: 2
m_micro_gcp.m_master.m_state: 2
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=1 0000000000000008]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
m_copyReason: 0 m_waiting: 0 0
c_copyGCISlave: sender{Data, Ref} 3 f60003 reason: 0 nextWord: 0
file[0] status: 2 type: 1 reqStatus: 0 file1: 2 1 0
c_nodeStartMaster.blockGcp: 0 4294967040
m_gcp_save.m_counter: 0 m_gcp_save.m_max_lag: 1220
m_micro_gcp.m_counter: 0 m_micro_gcp.m_max_lag: 1241
m_gcp_save.m_state: 0
m_gcp_save.m_master.m_state: 0
m_micro_gcp.m_state: 2
m_micro_gcp.m_master.m_state: 2
c_COPY_GCIREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_COPY_TABREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_CREATE_FRAGREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_DIH_SWITCH_REPLICA_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_EMPTY_LCP_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_COMMIT_Counter = [SignalCounter: m_count=1 0000000000000008]
c_GCP_PREPARE_Counter = [SignalCounter: m_count=0 0000000000000000]
c_GCP_SAVEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_INCL_NODEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_GCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_MASTER_LCPREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_INFOREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_START_RECREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_STOP_ME_REQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TC_CLOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
c_TCGETOPSIZEREQ_Counter = [SignalCounter: m_count=0 0000000000000000]
m_copyReason: 0 m_waiting: 0 0
c_copyGCISlave: sender{Data, Ref} 3 f60003 reason: 0 nextWord: 0
2009-10-15 02:33:54 [ndbd] INFO     -- Node 3 killed this node because GCP stop was detected
2009-10-15 02:33:54 [ndbd] INFO     -- NDBCNTR (Line: 263) 0x0000000a
2009-10-15 02:33:54 [ndbd] INFO     -- Error handler shutting down system
2009-10-15 02:33:54 [ndbd] INFO     -- Error handler shutdown completed - exiting
2009-10-15 02:33:56 [ndbd] ALERT    -- Node 3: Forced node shutdown completed. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.


node3 error log:
================
Time: Thursday 15 October 2009 - 02:33:54
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal e
rror, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 3 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 263) 0x0000000a
Program: ndbd
Pid: 21655
Trace: /DD3/ndb_3_trace.log.19
Version: mysql-5.1.35 ndb-7.0.7
***EOM***

