MySQL Bugs: #69457: Arbitration error

Bug #69457	Arbitration error
Submitted:	13 Jun 2013 7:30	Modified:	11 Apr 2016 13:40
Reporter:	Zhang Qi	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	7.2.6	OS:	Linux
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
Hi,

Our server down accidentally, here is NDB's error log, if you need trace log, we'll provide them.

Status: Temporary error, restart node
Message: Node declared dead. See error log for details (Arbitration error)
Error: 2315
Error data: We(5) have been declared dead by 4 (via 3) reason: Heartbeat failure(4)
Error object: QMGR (Line: 3934) 0x00000002
Program: ndbmtd
Pid: 28102 thr: 0
Version: mysql-5.5.22 ndb-7.2.6
Trace: /etc/mysql/mysql-cluster/ndb_data/ndb_5_trace.log.6 [t1..t7]

How to repeat:
Unknown, our server seems are doing join query.

Hello Zhang,

Most likely a saturated network, or overloaded CPUs issues. Node(5) died of heartbeat failure(Node 5 has been declared dead by 4 though node 3).
Could you please attach the cluster logs? Preferably using the ndb_error_reporter utility:

  http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-programs-ndb-error-reporter.html

Thanks,
Umesh

Hi Umesh,

Thanks for your quick responding.
I've uploaded the generated report in your ftp.
FILENAME: bug-data-69457.zip

Thanks,
Zhang

Hi Umesh,

Oops, we seems have a number of crash issues recently, you combine them together.
We frequently received data node down messages this year.
You can check the error log in ndb_5_error.log
Thanks ahead!
Zhang