version: mysql-5.6.14 ndb-7.3.3 cluster log: 2014-04-30 14:45:13 [MgmtSrvr] INFO -- Node 1: Local checkpoint 2617 completed 2014-04-30 14:57:08 [MgmtSrvr] INFO -- Node 1: Local checkpoint 2618 started. Keep GCI = 11910123 oldest restorable GCI = 11912322 2014-04-30 15:47:32 [MgmtSrvr] INFO -- Node 1: Local checkpoint 2618 completed 2014-04-30 15:47:33 [MgmtSrvr] INFO -- Node 1: Local checkpoint 2619 started. Keep GCI = 11913927 oldest restorable GCI = 11915936 2014-04-30 16:36:32 [MgmtSrvr] ALERT -- Node 1: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. 2014-04-30 16:36:32 [MgmtSrvr] ALERT -- Node 2: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. 2014-04-30 16:36:32 [MgmtSrvr] ALERT -- Node 50: Node 1 Disconnected 2014-04-30 16:36:32 [MgmtSrvr] ALERT -- Node 50: Node 2 Disconnected 2014-04-30 16:37:16 [MgmtSrvr] INFO -- Node 50: Node 2 Connected 2014-04-30 16:37:16 [MgmtSrvr] INFO -- Nodeid 2 allocated for NDB at 10.0.142.182 2014-04-30 16:37:22 [MgmtSrvr] INFO -- Node 2: Buffering maximum epochs 1000 2014-04-30 16:37:22 [MgmtSrvr] INFO -- Node 2: Communication to Node 1 opened 2014-04-30 16:37:22 [MgmtSrvr] INFO -- Node 2: Start phase 0 completed 2014-04-30 16:37:22 [MgmtSrvr] INFO -- Node 2: Waiting 30 sec for nodes 1 to connect, nodes [ all: 1 and 2 connected: 2 no-wait: ] 2014-04-30 16:37:25 [MgmtSrvr] INFO -- Node 2: Waiting 27 sec for nodes 1 to connect, nodes [ all: 1 and 2 connected: 2 no-wait: ] 2014-04-30 16:37:27 [MgmtSrvr] INFO -- Nodeid 1 allocated for NDB at 10.0.142.184 2014-04-30 16:37:28 [MgmtSrvr] INFO -- Node 2: Waiting 24 sec for nodes 1 to connect, nodes [ all: 1 and 2 connected: 2 no-wait: ] 2014-04-30 16:37:28 [MgmtSrvr] INFO -- Node 50: Node 1 Connected 2014-04-30 16:37:31 [MgmtSrvr] INFO -- Node 1: Buffering maximum epochs 1000 2014-04-30 16:37:31 [MgmtSrvr] INFO -- Node 1: CM_REGCONF president = 1, own Node = 1, our dynamic id = 0/1 2014-04-30 16:37:31 [MgmtSrvr] INFO -- Node 1: Communication to Node 2 opened 2014-04-30 16:37:31 [MgmtSrvr] INFO -- Node 1: Node 2 Connected 2014-04-30 16:37:31 [MgmtSrvr] INFO -- Node 1: Start phase 0 completed 2014-04-30 16:37:31 [MgmtSrvr] INFO -- Node 1: Start with all nodes 1 and 2 2014-04-30 16:37:31 [MgmtSrvr] INFO -- Node 1: Waiting 30 sec for nodes 2 to connect, nodes [ all: 1 and 2 connected: 1 no-wait: ] 2014-04-30 16:37:31 [MgmtSrvr] INFO -- Node 2: Node 1 Connected 2014-04-30 16:37:31 [MgmtSrvr] INFO -- Node 2: Waiting 21 sec for nodes 1 to connect, nodes [ all: 1 and 2 connected: 2 no-wait: ] 2014-04-30 16:37:34 [MgmtSrvr] INFO -- Node 1: Node 2: API mysql-5.6.14 ndb-7.3.3 2014-04-30 16:37:34 [MgmtSrvr] INFO -- Node 1: Start phase 1 completed 2014-04-30 16:37:34 [MgmtSrvr] INFO -- Node 1: Start phase 2 completed (system restart) 2014-04-30 16:37:34 [MgmtSrvr] INFO -- Node 2: CM_REGCONF president = 1, own Node = 2, our dynamic id = 0/2 2014-04-30 16:37:34 [MgmtSrvr] INFO -- Node 2: Node 1: API mysql-5.6.14 ndb-7.3.3 2014-04-30 16:37:34 [MgmtSrvr] INFO -- Node 2: Start phase 1 completed 2014-04-30 16:37:34 [MgmtSrvr] INFO -- Node 2: Start phase 2 completed (system restart) 2014-04-30 16:37:35 [MgmtSrvr] INFO -- Node 1: Restarting cluster to GCI: 11919740 2014-04-30 16:37:35 [MgmtSrvr] INFO -- Node 1: Starting to restore schema 2014-04-30 16:37:35 [MgmtSrvr] INFO -- Node 1: Start phase 3 completed (system restart) 2014-04-30 16:37:35 [MgmtSrvr] INFO -- Node 2: Start phase 3 completed (system restart) 2014-04-30 16:57:31 [MgmtSrvr] INFO -- Node 1: Restore of schema complete 2014-04-30 16:57:31 [MgmtSrvr] INFO -- Node 2: Starting to restore schema 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 10 done (sys/def/9/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 12 done (sys/def/11/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 14 done (sys/def/13/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 16 done (sys/def/15/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 18 done (sys/def/17/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 20 done (sys/def/19/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 22 done (sys/def/21/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 24 done (sys/def/23/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 26 done (sys/def/25/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 28 done (sys/def/27/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 30 done (sys/def/29/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 32 done (sys/def/31/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 34 done (sys/def/33/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 36 done (sys/def/35/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 38 done (sys/def/37/PRIMARY) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 39 done (sys/def/25/CSI_CustomerID) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 40 done (sys/def/25/CSI_InvoiceID) 2014-04-30 16:58:18 [MgmtSrvr] INFO -- Node 1: DICT: activate index 41 done (sys/def/25/CSI_StockItemID) ndb log: 2014-04-29 22:03:31 [ndbd] INFO -- Prepare arbitrator node 50 [ticket=0e5b002d1c34c84b] H'02c49943 H'00d03c00 scanIndex_abort state: 3 2014-04-30 16:36:32 [ndbd] INFO -- /export/home/pb2/build/sb_0-10782555-1384352627.99/mysql-cluster-gpl-7.3.3/storage/ndb/src/kernel/vm/SimulatedBlock.cpp 2014-04-30 16:36:32 [ndbd] INFO -- DBSPJ (Line: 1299) 0x00000002 2014-04-30 16:36:32 [ndbd] INFO -- Error handler shutting down system 2014-04-30 16:36:32 [ndbd] INFO -- Error handler shutdown completed - exiting 2014-04-30 16:36:32 [ndbd] ALERT -- Node 2: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. 2014-04-30 16:37:16 [ndbd] INFO -- Angel pid: 6355 started child: 6356 2014-04-30 16:37:16 [ndbd] INFO -- Configuration fetched from 'fps-db-c01:1186', generation: 1 ThreadConfig: input: main,ldm,ldm,recv,rep LockExecuteThreadToCPU: => parsed: main,ldm,ldm,recv,rep NDBMT: MaxNoOfExecutionThreads=5 NDBMT: workers=2 threads=2 tc=0 send=0 receive=1 2014-04-30 16:37:16 [ndbd] INFO -- NDB Cluster -- DB node 2 2014-04-30 16:37:16 [ndbd] INFO -- mysql-5.6.14 ndb-7.3.3 -- 2014-04-30 16:37:16 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : no numa support 2014-04-30 16:37:16 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 7382Mb initial: 7510Mb Adding 139Mb to ZONE_LO (1,4430) NDBMT: number of block threads=5 Instantiating DBSPJ instanceNo=0 Started thread, index = 0, id = 6358, type = SocketClientThread Started thread, index = 1, id = 6357, type = WatchDogThread Started thread, index = 2, id = 6359, type = SocketServerThread thr: 4 tid: 6356 (recv) THRMAN(5) TRPMAN(1) 2014-04-30 16:37:16 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Unknown place 0 elapsed=20 2014-04-30 16:37:16 [ndbd] INFO -- Watchdog: User time: 0 System time: 6 2014-04-30 16:37:16 [ndbd] WARNING -- Ndb kernel thread 1 is stuck in: Unknown place 0 elapsed=19 2014-04-30 16:37:16 [ndbd] INFO -- Watchdog: User time: 0 System time: 6 2014-04-30 16:37:16 [ndbd] WARNING -- Ndb kernel thread 2 is stuck in: Unknown place 0 elapsed=19 2014-04-30 16:37:16 [ndbd] INFO -- Watchdog: User time: 0 System time: 6 2014-04-30 16:37:16 [ndbd] WARNING -- Ndb kernel thread 3 is stuck in: Unknown place 0 elapsed=19 2014-04-30 16:37:16 [ndbd] INFO -- Watchdog: User time: 0 System time: 6 thr: 0 tid: 6360 (main) DBTC(0) DBDIH(0) DBDICT(0) NDBCNTR(0) QMGR(0) NDBFS(0) CMVMI(0) TRIX(0) DBUTIL(0) DBSPJ(0) THRMAN(0) TRPMAN(0) THRMAN(1) saving 0x7fa3b6980000 at 0xd2d290 (0) 2014-04-30 16:37:16 [ndbd] INFO -- Start initiated (mysql-5.6.14 ndb-7.3.3) thr: 2 tid: 6362 (ldm) PGMAN(1) DBACC(1) DBLQH(1) DBTUP(1) BACKUP(1) DBTUX(1) RESTORE(1) THRMAN(3) thr: 1 tid: 6361 (rep) BACKUP(0) DBLQH(0) DBACC(0) DBTUP(0) SUMA(0) DBTUX(0) TSMAN(0) LGMAN(0) PGMAN(0) RESTORE(0) DBINFO(0) PGMAN(3) THRMAN(2) thr: 3 tid: 6363 (ldm) PGMAN(2) DBACC(2) DBLQH(2) DBTUP(2) BACKUP(2) DBTUX(2) RESTORE(2) THRMAN(4) NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer Started thread, index = 8, id = 6364, type = NdbfsThread Adding 7373Mb to ZONE_LO (4431,235920) NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer Started thread, index = 9, id = 6365, type = NdbfsThread NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer Started thread, index = 10, id = 6366, type = NdbfsThread NDBFS/AsyncFile: Allocating 310392 for In/Deflate buffer Started thread, index = 11, id = 6367, type = NdbfsThread ~snip~ Started thread, index = 35, id = 6391, type = NdbfsThread Started thread, index = 36, id = 6392, type = NdbfsThread Started thread, index = 37, id = 6393, type = NdbfsThread WOPool::init(61, 9) RWPool::init(22, 14) WOPool::init(61, 9) RWPool::init(22, 14) RWPool::init(42, 75) 2014-04-30 16:37:20 [ndbd] INFO -- timerHandlingLab now: 2519163381 sent: 2519163185 diff: 196 LCPFragWatchdog : WarnPeriods : 2 MaxPeriods 6 : period millis : 10000 LCPFragWatchdog : WarnPeriods : 2 MaxPeriods 6 : period millis : 10000 blockSz: 800, wpp: 8188 -> 624 (76) RWPool::init(29, 624) RWPool::init(42, 16) RWPool::init(62, 12) RWPool::init(42, 83) 2014-04-30 16:37:21 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=100 2014-04-30 16:37:21 [ndbd] INFO -- Watchdog: User time: 105 System time: 414 2014-04-30 16:37:21 [ndbd] WARNING -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=200 2014-04-30 16:37:21 [ndbd] INFO -- Watchdog: User time: 111 System time: 418 2014-04-30 16:37:21 [ndbd] INFO -- timerHandlingLab now: 2519164378 sent: 2519163870 diff: 508 RWPool::init(c2, 18) RWPool::init(e2, 16) WOPool::init(41, 8) RWPool::init(82, 12) RWPool::init(a2, 54) WOPool::init(21, 10) WOPool::init(21, 10) WOPool::init(21, 10) blockSz: 1024, wpp: 8188 -> 816 (28) RWPool::init(68, 816) 2014-04-30 16:37:22 [ndbd] INFO -- Start phase 0 completed 2014-04-30 16:37:22 [ndbd] INFO -- Using locked memory 2014-04-30 16:37:34 [ndbd] INFO -- findNeighbours from: 2193 old (left: 65535 right: 65535) new (1 1) saving 0x7fa3b6ac0000 at 0xd43290 (0) saving 0x7fa3b6b60000 at 0xd4e290 (0) Dbspj::execSTTOR() inst:0 phase=1 2014-04-30 16:37:34 [ndbd] INFO -- Start phase 1 completed 2014-04-30 16:37:34 [ndbd] INFO -- Start phase 2 completed saving 0x7fa3b6a20000 at 0xd38290 (0) 2014-04-30 16:37:35 [ndbd] INFO -- Start phase 3 completed Found pending trans (0) - committing commit create 599 commit create 600 restartCreateObj(1) file: 1 restartCreateObj(821) file: 1 restartCreateObj(2) file: 1 restartCreateObj(3) file: 1 restartCreateObj(4) file: 1 restartCreateObj(5) file: 1 restartCreateObj(6) file: 1 restartCreateObj(7) file: 1 restartCreateObj(9) file: 1 restartCreateObj(11) file: 1 mysqld log: 2014-04-30 15:59:56 21563 [Note] NDB Binlog: CREATE TABLE Event: REPL$fps_demo/template_part_type_field 2014-04-30 16:36:32 21563 [Note] NDB Binlog: Node: 1, down, Subscriber bitmask 00 2014-04-30 16:36:32 21563 [Note] NDB Binlog: Node: 2, down, Subscriber bitmask 00 2014-04-30 16:36:32 21563 [Note] NDB Binlog: cluster failure for ./mysql/ndb_schema at epoch 11919742/0. 2014-04-30 16:36:32 21563 [ERROR] Got error 4010 when reading table './fps_850176b/lot' 2014-04-30 16:36:33 21563 [Note] NDB Binlog: cluster failure for ./mysql/ndb_apply_status at epoch 11919742/0. 2014-04-30 16:36:33 21563 [Note] Restarting Cluster Binlog 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/supplier.frm' 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_850176b/lot.frm' 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_850155/lot_image_queue.frm' 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_850155/lot.frm' 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_850155/lot_image_queue.frm' 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_850155/lot.frm' 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_850155/lot_image_queue.frm' 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_850155/lot.frm' 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/lot_image_queue.frm' 2014-04-30 16:36:33 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/lot.frm' 2014-04-30 16:36:34 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/lot_image_queue.frm' 2014-04-30 16:36:34 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/lot.frm' 2014-04-30 16:36:34 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/lot_image_queue.frm' 2014-04-30 16:36:34 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/lot.frm' 2014-04-30 16:36:35 21563 [Warning] NDB: Could not acquire global schema lock (4009)Cluster Failure 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/barcode_type.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/batch.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/batch_lot.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/box.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/box_argument.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/buyer.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/carrier.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/carrier_fust.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/cbs_group.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/cbs_group_data.frm' 2014-04-30 16:36:35 21563 [ERROR] /usr/local/mysql/bin/mysqld: Incorrect information in file: './fps_demo/cites_product_group.frm' config.ini: [NDB_MGMD DEFAULT] Portnumber=1186 [NDB_MGMD] NodeId=50 HostName=fps-db-c01 DataDir=/var/lib/mysql-cluster/50/ Portnumber=1186 [NDB_MGMD] NodeId=51 HostName=fps-db-c03 DataDir=/var/lib/mysql-cluster/51/ Portnumber=1186 [TCP DEFAULT] SendBufferMemory=4M ReceiveBufferMemory=4M [NDBD DEFAULT] DataMemory=6G DiskCheckPointSpeed=10M DiskCheckPointSpeedInRestart=100M IndexMemory=1G LockPagesInMainMemory=1 MaxNoOfAttributes=1000000 MaxNoOfConcurrentOperations=1000000 MaxBufferedEpochs=1000 MaxNoOfExecutionThreads=4 MaxNoOfOrderedIndexes=50000 MaxNoOfTables=8000 RedoBuffer=32M TimeBetweenEpochs=200 TimeBetweenGlobalCheckpoints=1000 [NDBD] NodeId=2 HostName=fps-db-c02 DataDir=/var/lib/mysql-cluster/2/ [NDBD] NodeId=1 HostName=fps-db-c04 DataDir=/var/lib/mysql-cluster/1/ [MYSQLD DEFAULT] [MYSQLD] NodeId=60 HostName=fps-db-c01 [MYSQLD] NodeId=62 HostName=fps-db-c01 [MYSQLD] NodeId=61 HostName=fps-db-c03 [MYSQLD] NodeId=63 HostName=fps-db-c03