MySQL Bugs: #72784: Internal program error (failed ndbrequire) During Rolling Restart

Bug #72784	Internal program error (failed ndbrequire) During Rolling Restart
Submitted:	28 May 2014 13:23	Modified:	25 Feb 2016 14:05
Reporter:	Joel Hanger	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	-7.3.5-1.el6.x86_64	OS:	Linux (CentOS release 6.5 - kernel 2.6.32-431.el6.centos.plus.x86_64)
Assigned to:	MySQL Verification Team	CPU Architecture:	Any
Tags:	2341, crash, data, ndb-7.3.5, ndbmtd, node, QmgrMain.cpp, restart, rolling

Description:
While adding 2 data nodes to the cluster I performed a rolling restart, or at least attempted. The first data node fails to restart and gives error 2341. 

Shutting down the cluster and restarting does work, however this isn't feasible for when we move this to a production system.

This is done on an amazon VM setup. 

Data Nodes:
  r3.xlarge:
     4 core, 30.5GB ram, 8 stripe ebs volume @ 200GB 
     using ndbmtd

API Nodes:
  m3.xlarge (moving to r3.xlarge)
     4 core, 15GB ram, 8 stripe ebs volume @ 200GB

How to repeat:
Setup cluster 7.3.5-1
2 data node groups
add data to full (reached peak capacity loading data for testing)

add node group, restart management server (--reload)
perform rolling restart, errors on first node restart

I don't know if it's related or not to:
http://bugs.mysql.com/bug.php?id=65084

Hi

I cannot reproduce this problem with Oracle binaries and yes, I can reproduce it with RH binaries. Looking at your report you are using RHEL binaries, I strongly suggest you use Oracle made binaries. For RHEL if you are using community build you can setup YUM repository for your RHEL and install Oracle binaries just like you would use the RH prepared ones. If you want you can download RPM's and upgrade manually. If you want to use Enteprise binaries check with support team how to set it up best for your system.

Best regards
Bogdan Kecman