Bug #70180 Ndb dead for Heartbeat failure.
Submitted: 29 Aug 2013 4:32 Modified: 14 Jun 2016 13:03
Reporter: rz J Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:7.2.10 OS:Linux (RHEL6)
Assigned to: MySQL Verification Team CPU Architecture:Any
Tags: cluster, heartbeat, ndb

[29 Aug 2013 4:32] rz J 
Description:
hi, 
Our server down for Heartbeat failure.
here is report:
Time: Thursday 29 August 2013 - 11:03:08
Status: Temporary error, restart node
Message: Node declared dead. See error log for details (Arbitration error)
Error: 2315
Error data: We(16) have been declared dead by 23 (via 19) reason: Heartbeat failure(4)
Error object: QMGR (Line: 3934) 0x00000002
Program: ndbmtd
Pid: 14599 thr: 0
Version: mysql-5.5.29 ndb-7.2.10
Trace: /data/mysql/data/ndb_16_trace.log.10 [t1..t4]
***EOM***

any idea for this?

How to repeat:
Unknown
[30 Aug 2013 7:38] Hartmut Holzgraefe 
Hard to tell without seeing the cluster log contents, but usually a heartbeat failure is not a bug but "just" a network problem that caused the node to not reply to heartbeat messages in time for four (or was it three?) times in a row.
Can also be caused by extremely high data node load ...
[2 Sep 2013 1:02] rz J 
Thank you, Hartmut.
I cannot access the FTP with my company's limit, maybe upload later.
But the horrible thing is after cluster restart, all data ware lost!