MySQL Bugs: #79702: Node 1: Forced node shutdown completed. Caused by error 6050: 'WatchDog terminat

Bug #79702	Node 1: Forced node shutdown completed. Caused by error 6050: 'WatchDog terminat
Submitted:	18 Dec 2015 12:13	Modified:	30 Jan 2016 13:59
Reporter:	Kornchai Chawanrattana	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:		OS:	Ubuntu (14.04)
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
I have mysql cluster with 2 data node one of them (node id: 1)
shutdown so offen, and it occured on the same node and I guest it not because of work load right now my server have only a few active users less than 100 per day

##### command ####### free -m ##########
total       used       free     shared    buffers     cached
Mem:          3953       3530        423          0        198        560
-/+ buffers/cache:       2771       1182
Swap:            0          0          0

######### ndb_1_error.log ############

Time: Friday 18 December 2015 - 00:18:47
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19292) 0x00000002
Program: ndbmtd
Pid: 22604 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.4 [t1..t4]
***EOM***
                                                                   
Time: Friday 18 December 2015 - 02:28:55
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19292) 0x00000002
Program: ndbmtd
Pid: 23193 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.5 [t1..t4]
***EOM***
                                                                   
Time: Friday 18 December 2015 - 06:41:28
Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine running this node (Internal error, programming error or missing error message, please report a bug)
Error: 6050
Error data: Job Handling
Error object: /export/home/pb2/build/sb_0-16730888-1444648756.12/mysql-cluster-gpl-7.4.8/storage/ndb/src/kernel/vm/WatchDog.cpp
Program: ndbmtd
Pid: 23426
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/
Time: Friday 18 December 2015 - 06:41:28
Status: Temporary error, restart node
Message: Node declared dead. See error log for details (Arbitration error)
Error: 2315
Error data: We(1) have been declared dead by 2 (via 2) reason: Heartbeat failure(4)
Error object: QMGR (Line: 4213) 0x00000002
Program: ndbmtd
Pid: 23426 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.7 [t1..t4]
***EOM*** 
                                                                 
Time: Thursday 17 December 2015 - 12:03:10
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19292) 0x00000002
Program: ndbmtd
Pid: 19824 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.18 [t1..t4]
***EOM***
                                                                
Time: Thursday 17 December 2015 - 12:14:04
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19292) 0x00000002
Program: ndbmtd
Pid: 20033 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.19 [t1..t4]
***EOM***
                                                                
Time: Thursday 17 December 2015 - 12:33:42
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19292) 0x00000002
Program: ndbmtd
Pid: 20206 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.20 [t1..t4]
***EOM***
                                                                
Time: Thursday 17 December 2015 - 12:41:27
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19292) 0x00000002
Program: ndbmtd
Pid: 20320 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.21 [t1..t4]
***EOM***
                                                                
Time: Thursday 17 December 2015 - 12:49:17
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19292) 0x00000002
Program: ndbmtd
Pid: 20419 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.22 [t1..t4]
***EOM***
                                                                
Time: Thursday 17 December 2015 - 13:19:33
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19292) 0x00000002
Program: ndbmtd
Pid: 20630 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.23 [t1..t4]
***EOM***
                                                                
Time: Thursday 17 December 2015 - 13:25:29
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtcMain.cpp
Error object: DBTC (Line: 19292) 0x00000002
Program: ndbmtd
Pid: 20819 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.24 [t1..t4]
***EOM***
                                                                
Time: Thursday 17 December 2015 - 13:44:22
Status: Temporary error, restart node
Message: Another node failed during system restart, please investigate error(s) on other node(s) (Restart error)
Error: 2308
Error data: Node 2 disconnected
Error object: QMGR (Line: 3473) 0x00000002
Program: ndbmtd
Pid: 21033 thr: 0
Version: mysql-5.6.27 ndb-7.4.8
Trace: /var/lib/mysql-cluster/log/ndb_1_trace.log.25 [t1..t4]
***EOM***

How to repeat:

It occurred offen but now always

Hi,

Is it always the same data node you have issues with?
Can you upload full trace logs (use ndb_error_reporter script to collect all logs)?

all best
Bogdan Kecman

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".