Bug #66525 Nodes are stopped with Internal program error (failed ndbrequire) 2341
Submitted: 24 Aug 2012 10:17 Modified: 21 Nov 2016 12:37
Reporter: Attila Kanto Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:7.2.6 OS:Linux (CentOS 6.2, Kernel: 2.6.32-220.17.1.el6.x86_64)
Assigned to: Bogdan Kecman CPU Architecture:Any
Tags: DblqhMain.cpp, failed ndbrequire, internal program error, restart node, Temporary error

[24 Aug 2012 10:17] Attila Kanto
Description:
In MySQL cluster version mysql-5.5.22 ndb-7.2.6, the Nodes are failing sporadically with the following error message: "Internal error, programming error or missing error message, please report a bug". Although the  "Temporary error, restart node" error message present in the logs, the Nodes are not restarted properly and the cluster is unable to recover.

The logs shows the following error messages:

ALERT    -- Node 11: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. 

Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 9785) 0x00000006
Program: ndbd
Pid: 2655
Version: mysql-5.5.22 ndb-7.2.6
Trace: /u01/local/mysql/data/ndb_10_trace.log.1 [t1..t1] 

Detailed logs are available in the output of ndb_error_report in attachment.

How to repeat:
The issue occurs sporadically and there are no exact steps to reproduce the issue
[24 Aug 2012 10:19] Attila Kanto
Output of ndb_error_reporter  /u01/local/mysql-cluster/config.ini

Attachment: ndb_error_report_20120824110329.tar.bz2 (application/octet-stream, text), 57.84 KiB.

[19 Sep 2012 0:14] Frank Zhou
I got the exactly error, repeatable. Please help to fix it or provide work-around.
[19 Sep 2012 7:25] Attila Kanto
Probably this issue is related to the followings:

http://bugs.mysql.com/bug.php?id=65667
http://bugs.mysql.com/bug.php?id=65084

We have upgraded to MySQL Cluster 7.2.7 version and it seems that this sporadic shutdown does not occur anymore.
[24 Oct 2012 13:58] Brian Hobson
We're having the exact same error sporadically with 7.2.7, so 7.2.7 is NOT the fix.
[18 Jan 2013 21:14] Shahryar Ghazi
Hi,

It looks like this bug report is a duplicate of Bug # 65084 which has already been fixed in version 7.2.7. Please upgrade to version 7.2.7 to avoid this bug.

@Brian: Are you having the exact same issue? ie. error messages, file name(s) and line numbers etc and problem description are identical. Please create a new bug report if your issue is different so I could close this bug else provide more details about your configuration. For example, config files (config.ini, my.cnf), ndb error report or log files, steps to reproduce the issue, exact version number and anything that might be helpful to easily reproduce the issue.

@Frank: Did you upgrade to 7.2.7?

Please let me know.

Thanks.
[19 Feb 2013 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[6 Mar 2013 15:50] Zhihong He
Hi,

I am using mysql-5.5.25 ndb-7.2.7, and had exactly the same problem when using ALTER ONLINE TABLE ... REORGANIZE PARTITION; to a 1 million lines table.

 
After restart the cluster, seems the table is correctly reorganized.
[8 Mar 2013 18:42] Shahryar Ghazi
@Zhihong He: Can you try and reproduce the issue and let us know the exact steps please?
[14 Mar 2013 1:49] Zhihong He
Hi, after updated to 5.5.29-7.2.10, when using:
ALTER ONLINE TABLE table_name REORGANIZE PARTITION 
to a 7 million rows table (4 partitions and 1+ million lines each ), it causes the cluster crash and without any log info. 
When restart cluster, then I got this issue. Before, when I using 5.5.25-7.2.7 version, I can restart the cluster after trying 2-3 times. But with this new version, I have no luck and always get this error.
[21 Nov 2016 12:37] Bogdan Kecman
Hi,

I cannot reproduce this problem on any of the recent 7.2, 7.3 and 7.4 versions

regards
Bogdan Kecman