Bug #72784 Internal program error (failed ndbrequire) During Rolling Restart
Submitted: 28 May 2014 13:23 Modified: 25 Feb 2016 14:05
Reporter: Joel Hanger Email Updates:
Status: Not a Bug Impact on me:
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:-7.3.5-1.el6.x86_64 OS:Linux (CentOS release 6.5 - kernel 2.6.32-431.el6.centos.plus.x86_64)
Assigned to: MySQL Verification Team CPU Architecture:Any
Tags: 2341, crash, data, ndb-7.3.5, ndbmtd, node, QmgrMain.cpp, restart, rolling

[28 May 2014 13:23] Joel Hanger
While adding 2 data nodes to the cluster I performed a rolling restart, or at least attempted. The first data node fails to restart and gives error 2341. 

Shutting down the cluster and restarting does work, however this isn't feasible for when we move this to a production system.

This is done on an amazon VM setup. 

Data Nodes:
     4 core, 30.5GB ram, 8 stripe ebs volume @ 200GB 
     using ndbmtd

API Nodes:
  m3.xlarge (moving to r3.xlarge)
     4 core, 15GB ram, 8 stripe ebs volume @ 200GB

How to repeat:
Setup cluster 7.3.5-1
2 data node groups
add data to full (reached peak capacity loading data for testing)

add node group, restart management server (--reload)
perform rolling restart, errors on first node restart
[28 May 2014 13:26] Joel Hanger
I don't know if it's related or not to:
[25 Feb 2016 14:05] MySQL Verification Team

I cannot reproduce this problem with Oracle binaries and yes, I can reproduce it with RH binaries. Looking at your report you are using RHEL binaries, I strongly suggest you use Oracle made binaries. For RHEL if you are using community build you can setup YUM repository for your RHEL and install Oracle binaries just like you would use the RH prepared ones. If you want you can download RPM's and upgrade manually. If you want to use Enteprise binaries check with support team how to set it up best for your system.

Best regards
Bogdan Kecman