MySQL Bugs: #62413: MySQL Cluster data node daemon shutdown issue

Bug #62413	MySQL Cluster data node daemon shutdown issue
Submitted:	12 Sep 2011 15:26	Modified:	19 Oct 2016 23:03
Reporter:	Kumar Vikram Dev	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	gpl-7.1.13-linux-x86_64-glibc23	OS:	Linux (2.6.18-164.el5xen)
Assigned to:		CPU Architecture:	Any

Description:
Hi,
We are facing frequent problems of mysql Cluster data node daemon as it keeps shutting down on its own after running for a day or two.

====================

Time: Saturday 10 September 2011 - 14:35:37

Status: Temporary error, restart node

Message: WatchDog terminate, internal error or massive overload on the machine running this node (Internal error, programming error or missing error message, please report a bug)

Error: 6050

Error data: Job Handling

Error object: WatchDog.cpp

Program: ndbd

Pid: 25466

Version: mysql-5.1.56 ndb-7.1.13

Trace: /home/mysql/data/data1/ndb_11_trace.log.4

***EOM***

====================

 

 

2011-09-10 13:39:51 [MgmtSrvr] WARNING  -- Node 11: Node 12 missed heartbeat 2

2011-09-10 13:39:52 [MgmtSrvr] WARNING  -- Node 11: Node 12 missed heartbeat 3

2011-09-10 13:39:52 [MgmtSrvr] ALERT    -- Node 1: Node 12 Disconnected

2011-09-10 13:39:54 [MgmtSrvr] WARNING  -- Node 11: Node 12 missed heartbeat 4

2011-09-10 13:39:54 [MgmtSrvr] ALERT    -- Node 11: Node 12 declared dead due to missed heartbeat

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: Communication to Node 12 closed

2011-09-10 13:39:54 [MgmtSrvr] ALERT    -- Node 11: Network partitioning - arbitration required

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: President restarts arbitration thread [state=7]

2011-09-10 13:39:54 [MgmtSrvr] ALERT    -- Node 11: Arbitration won - positive reply from node 1

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: GCP Take over started

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: Node 11 taking over as DICT master

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: GCP Monitor: Computed max GCP_SAVE lag to 131 seconds

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: GCP Monitor: Computed max GCP_COMMIT lag to 13 seconds

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: GCP Take over completed

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: kk: 2579228/1185 0 1

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: LCP Take over started

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: ParticipatingDIH = 0000000000000000

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: ParticipatingLQH = 0000000000000000

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: m_LCP_COMPLETE_REP_Counter_DIH = [SignalCounter: m_count=0 0000000000000000]

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: m_LCP_COMPLETE_REP_Counter_LQH = [SignalCounter: m_count=0 0000000000000000]

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: m_LAST_LCP_FRAG_ORD = [SignalCounter: m_count=0 0000000000000000]

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: m_LCP_COMPLETE_REP_From_Master_Received = 1

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: LCP Take over completed (state = 4)

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: ParticipatingDIH = 0000000000000000

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: ParticipatingLQH = 0000000000000000

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: m_LCP_COMPLETE_REP_Counter_DIH = [SignalCounter: m_count=0 0000000000000000]

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: m_LCP_COMPLETE_REP_Counter_LQH = [SignalCounter: m_count=0 0000000000000000]

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: m_LAST_LCP_FRAG_ORD = [SignalCounter: m_count=0 0000000000000000]

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: m_LCP_COMPLETE_REP_From_Master_Received = 1

2011-09-10 13:39:54 [MgmtSrvr] ALERT    -- Node 11: Node 12 Disconnected

2011-09-10 13:39:54 [MgmtSrvr] INFO     -- Node 11: Started arbitrator node 1 [ticket=637a00023622c997]

2011-09-10 13:39:57 [MgmtSrvr] INFO     -- Node 11: Communication to Node 12 opened

2011-09-10 13:41:20 [MgmtSrvr] ALERT    -- Node 12: Forced node shutdown completed. Occured during startphase 0. Initiated by signal 11.

2011-09-10 13:41:21 [MgmtSrvr] INFO     -- Mgmt server state: nodeid 12 freed, m_reserved_nodes 1, 11, 111 and 112.

2011-09-10 14:33:04 [MgmtSrvr] WARNING  -- Node 11: GCP Monitor: GCP_SAVE lag 60 seconds (max lag: 131s)

2011-09-10 14:34:08 [MgmtSrvr] WARNING  -- Node 11: GCP Monitor: GCP_SAVE lag 120 seconds (max lag: 131s)

2011-09-10 14:34:26 [MgmtSrvr] ALERT    -- Node 1: Node 11 Disconnected

2011-09-10 14:35:38 [MgmtSrvr] ALERT    -- Node 11: Forced node shutdown completed. Occured during startphase 0. Initiated by signal 11.

2011-09-10 14:35:39 [MgmtSrvr] INFO     -- Mgmt server state: nodeid 11 freed, m_reserved_nodes 1, 111 and 112.

How to repeat:
None

Suggested fix:
None

Hi,
I have a report file which is around 1MB and I am unable to access ftp://ftp.mysql.com/pub/mysql/upload/. Please suggest how do I pass it on to you.

Thanks,
Vikram

Hi,
Can someone have a look at this issue and suggest some solution asap.

Thanks,
Vikram

same issue

Attachment: mysql_cluster_error_info.txt (text/plain), 9.99 KiB.

I am facing the same issue with mysql-cluster-gpl-7.3.3-linux-glibc2.5-x86_64.

It is incredibly frustrating yet typical that a bug report goes completely ignored by the MySQL/Oracle team, despite the documentation asking for a bug report to be filed.

Frequently this is caused the the disk subsystem becoming unresponsive. Prior to 7.1.10 the behavior of cluster was to trigger a node failure when the GCP_commit lag exceeded 4 second. This lead to a situation where temporary lags due to large commits could trigger an unwanted node failure that could be avoided by sleeping briefly. Starting with 7.2 we instead disabled the GCP_commit lag timeout altogether by setting TimeBetweenEpochsTimeout = 0. To trigger a lagging node to fail automatically when it is unable to participate in GCP you must set TimeBetweenEpochsTimeout, a good value to start with is 30000( 30s ).