Bug #69072 Internal program error (failed ndbrequire) error:2341
Submitted: 25 Apr 2013 18:01 Modified: 2 Jun 2013 6:29
Reporter: Ben Im Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:7.3.1 OS:Linux (ubuntu 12.0.4)
Assigned to: CPU Architecture:Any

[25 Apr 2013 18:01] Ben Im
Description:
NDB data node went down with error below. All data nodes were functioning fine. And then below error popped up. Putting error, out, and trace logs on ftp.oracle.com under support/incoming. The files are

-rw-r--r-- 1 root root    1566 Apr 24 15:41 ndb_4_error.log
-rw-r--r-- 1 root root   88859 Apr 24 16:49 ndb_4_out.log
-rw-r--r-- 1 root root 1056816 Apr 24 15:41 ndb_4_trace.log.3

2013-04-22 16:00:10 [ndbd] ALERT    -- Node 4: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

Setup below:

ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     6 node(s)
id=3    @192.168.104.42  (mysql-5.6.10 ndb-7.3.1, Nodegroup: 0, Master)
id=4    @192.168.104.43  (mysql-5.6.10 ndb-7.3.1, Nodegroup: 0)
id=5    @192.168.104.44  (mysql-5.6.10 ndb-7.3.1, Nodegroup: 1)
id=6    @192.168.104.121  (mysql-5.6.10 ndb-7.3.1, Nodegroup: 1)
id=7    @192.168.104.122  (mysql-5.6.10 ndb-7.3.1, Nodegroup: 2)
id=8    @192.168.104.46  (mysql-5.6.10 ndb-7.3.1, Nodegroup: 2)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @192.168.104.48  (mysql-5.6.10 ndb-7.3.1)

[mysqld(API)]   7 node(s)
id=20   @192.168.104.42  (mysql-5.6.10 ndb-7.3.1)
id=21   @192.168.104.43  (mysql-5.6.10 ndb-7.3.1)
id=22   @192.168.104.44  (mysql-5.6.10 ndb-7.3.1)
id=23   @192.168.104.121  (mysql-5.6.10 ndb-7.3.1)
id=24   @192.168.104.122  (mysql-5.6.10 ndb-7.3.1)
id=25   @192.168.104.46  (mysql-5.6.10 ndb-7.3.1)
id=26 (not connected, accepting connect from any host)

How to repeat:
No steps to reproduce. All data nodes were in light use, and then node 4 went down.
[30 Apr 2013 10:03] Gustaf Thorslund
Hi Ben,

Could you please attach the output from ndb_error_reporter without the --fs option? See:

  http://dev.mysql.com/doc/refman/5.6/en/mysql-cluster-programs-ndb-error-reporter.html

It is often useful to start looking at the cluster log and also look at the other node in the same node group as the failing node. Then there are trace/error files too included by ndb_error_reporter. Since they report will be compressed it will minimise the size of file to attach.

/Gustaf
[30 Apr 2013 15:54] Ben Im
FTP'ed ndb_error_report_20130430084844.tar.bz2 in support/incoming directory. Let me know how it looks.
[2 May 2013 6:29] Gustaf Thorslund
Hi Ben,

I'm sorry to say, but it looks kind of empty. This is what I see on the server:
  ndb_error_report_20130430084844.tar.bz2 	1 KB 	2013-04-30 	15:51:00

Downloading and looking inside I only see:
$ tar tjf ndb_error_report_20130430084844.tar.bz2 
ndb_error_report_20130430084844/

Did you get any errors when running ndb_error_reporter?

Can you ssh into the different hosts from the data nodes from the management node?

Does the user you invoke ndb_error_reporter with have read access to the logs?

/Gustaf
[3 Jun 2013 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".