MySQL Bugs: #66525: Nodes are stopped with Internal program error (failed ndbrequire) 2341

Bug #66525	Nodes are stopped with Internal program error (failed ndbrequire) 2341
Submitted:	24 Aug 2012 10:17	Modified:	21 Nov 2016 12:37
Reporter:	Attila Kanto	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	7.2.6	OS:	Linux (CentOS 6.2, Kernel: 2.6.32-220.17.1.el6.x86_64)
Assigned to:	MySQL Verification Team	CPU Architecture:	Any
Tags:	DblqhMain.cpp, failed ndbrequire, internal program error, restart node, Temporary error

Description:
In MySQL cluster version mysql-5.5.22 ndb-7.2.6, the Nodes are failing sporadically with the following error message: "Internal error, programming error or missing error message, please report a bug". Although the  "Temporary error, restart node" error message present in the logs, the Nodes are not restarted properly and the cluster is unable to recover.

The logs shows the following error messages:

ALERT    -- Node 11: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. 

Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 9785) 0x00000006
Program: ndbd
Pid: 2655
Version: mysql-5.5.22 ndb-7.2.6
Trace: /u01/local/mysql/data/ndb_10_trace.log.1 [t1..t1] 

Detailed logs are available in the output of ndb_error_report in attachment.

How to repeat:
The issue occurs sporadically and there are no exact steps to reproduce the issue

Output of ndb_error_reporter /u01/local/mysql-cluster/config.ini

Attachment: ndb_error_report_20120824110329.tar.bz2 (application/octet-stream, text), 57.84 KiB.

I got the exactly error, repeatable. Please help to fix it or provide work-around.

Probably this issue is related to the followings:

http://bugs.mysql.com/bug.php?id=65667
http://bugs.mysql.com/bug.php?id=65084

We have upgraded to MySQL Cluster 7.2.7 version and it seems that this sporadic shutdown does not occur anymore.

We're having the exact same error sporadically with 7.2.7, so 7.2.7 is NOT the fix.

Hi,

It looks like this bug report is a duplicate of Bug # 65084 which has already been fixed in version 7.2.7. Please upgrade to version 7.2.7 to avoid this bug.

@Brian: Are you having the exact same issue? ie. error messages, file name(s) and line numbers etc and problem description are identical. Please create a new bug report if your issue is different so I could close this bug else provide more details about your configuration. For example, config files (config.ini, my.cnf), ndb error report or log files, steps to reproduce the issue, exact version number and anything that might be helpful to easily reproduce the issue.

@Frank: Did you upgrade to 7.2.7?

Please let me know.

Thanks.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Hi,

I am using mysql-5.5.25 ndb-7.2.7, and had exactly the same problem when using ALTER ONLINE TABLE ... REORGANIZE PARTITION; to a 1 million lines table.

 
After restart the cluster, seems the table is correctly reorganized.

@Zhihong He: Can you try and reproduce the issue and let us know the exact steps please?

Hi, after updated to 5.5.29-7.2.10, when using:
ALTER ONLINE TABLE table_name REORGANIZE PARTITION 
to a 7 million rows table (4 partitions and 1+ million lines each ), it causes the cluster crash and without any log info. 
When restart cluster, then I got this issue. Before, when I using 5.5.25-7.2.7 version, I can restart the cluster after trying 2-3 times. But with this new version, I have no luck and always get this error.

Hi,

I cannot reproduce this problem on any of the recent 7.2, 7.3 and 7.4 versions

regards
Bogdan Kecman