MySQL Bugs: #65012: Data nodes crash continuously after upgrade from 7.2.4 to 7.2.5

Bug #65012	Data nodes crash continuously after upgrade from 7.2.4 to 7.2.5
Submitted:	17 Apr 2012 18:20	Modified:	24 Apr 2012 6:15
Reporter:	Chris Miller	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	7.2.5	OS:	Linux (RedHat 6.2)
Assigned to:		CPU Architecture:	Any

Description:

Our cluster was upgraded from 7.2.4 to 7.2.5. Immediately after, each of the data nodes would crash within one hour. Restarting the first crashed data node seemed to keep the cluster alive, but both nodes would crash without attention.

Attempts were made to refresh via a rolling restart using 'ndbd --initial' to refresh the data nodes. Ultimately the cluster had to be downgraded to 7.2.4 to restore service. Restarts seem to hang during Phase 5 of the startup sequence.

We are running the single threaded ndbd on Xeon 5600 series VMware servers with 2 virtual CPUs. 8GB of memory are assigned to each data node, with approximately 5GB used by ndbd, and always 1GB of memory free. The servers are interconnected with 10Gb ethernet.

How to repeat:
Upgrade existing cluster from 7.2.4 to 7.2.5. Wait 10-60 minutes and one node will crash, followed by the other within an hour.

Unable to attach error report due to 500k max filesize

Do you have any place where you could store the files and paste the link?

After additional testing across two clusters, it appears this issue is not related to an upgrade. It appears this issue exists in both version 7.2.4 and 7.2.5, and is triggered when the cluster is under a mild load from 3-4 web application nodes. I will open up a new bug with the specifics.

Opened new bug here :

http://bugs.mysql.com/bug.php?id=65084