MySQL Bugs: #61879: MySQL Cluster randomly crashes

Bug #61879	MySQL Cluster randomly crashes
Submitted:	15 Jul 2011 15:54	Modified:	2 Apr 2018 14:13
Reporter:	Léo Peltier	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Cluster: Disk Data	Severity:	S2 (Serious)
Version:	mysql-5.1.39 ndb-7.0.9b	OS:	Linux (2.6.32-311-ec2 #23-Ubuntu SMP Thu Dec 2 11:14:35 UTC 2010 x86_64 GNU/Linux)
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
On a two data nodes cluster, the second node randomly crashes.
The nodes are hosted on cloud instances with poor disk i/o so the CPU load is usually around 2.0 (mostly iowait) but a minute or two before a crash occurs the load goes as high as 3 on the first node and 5~15 on the second one.

Trace files are empty.

How to repeat:
Randomly happens. I did not find anything that could explain those crashes (no traffic spike or anything like that).

Suggested fix:
Restarting the failing node works fine for a while.

ndb_5_out.log

Attachment: ndb_5_out.log (text/x-log), 366.67 KiB.

ndb_5_error.log

Attachment: ndb_5_error.log (text/x-log), 1.95 KiB.

Hi,

That system is just not capable of sustaining required load and cluster behaves as designed (it will crash rather then fail in delivering configured troughput).

Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine 

Status: Temporary error, restart node
Message: WatchDog terminate, internal error or massive overload on the machine

Please contact our support team to help you how to try to configure cluster to somehow run on that environment but you have to understand that while our support and consulting teams do regularly perform miracles some things are out of our reach :(

all best
Bogdan