MySQL Bugs: #61905: Error: 2341 MySQL Cluster Crash On Loading SQL Insert Statements

Bug #61905	Error: 2341 MySQL Cluster Crash On Loading SQL Insert Statements
Submitted:	19 Jul 2011 4:30	Modified:	16 Aug 2011 9:16
Reporter:	Bill Kunyiha	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	MySQL Cluster 7.1.15	OS:	Linux
Assigned to:	Assigned Account	CPU Architecture:	Any
Tags:	Error: 2341 Internal program error (failed ndbrequire) (Internal error, please report a bug), programming error or missing error message

Description:
We have two MySQL Cluster management servers, two NDB Servers and 4 SQL servers. 
I installed MySQL cluster and now trying to upload Insert statements from our other Non Cluster MySQL Server and getting the following error 

The two NDB servers have IP’s 192.168.10.114 and 192.168.10.115

When I run check-ndbd-errlog.sh from the management server I get the following error.

[root@webportal01 tools]# ./check-ndbd-errlog.sh 
select data node to view error log from
id         data node
3          192.168.10.114
4          192.168.10.115
Select an id above: 4
Current byte-offset of file-pointer is: 1566                      

Time: Sunday 17 July 2011 - 13:20:07
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtup/DbtupRoutines.cpp
Error object: DBTUP (Line: 669) 0x00000002
Program: /usr/sbin//ndbmtd
Pid: 9211 thr: 2
Version: mysql-5.1.56 ndb-7.1.15
Trace: /data/mysqlcluster//ndb_4_trace.log.3 /data/mysqlcluster//ndb_4_trace.log.3_t1 /data/mysqlcluster//ndb_4_trace.log

How to repeat:
Restart Cluster and try loading the Insert statements.

Whole Cluster shuts down.

Hi,
I also got this error same, pls help.

I only have this problem with MySQL Cluster 7.1  and NOT MySQL Cluster 7.0.
So if you need your cluster up and running soon, I would recommend you use MySQL Cluster 7.0  

Go to http://www.severalnines.com/resources/configurator

SELECT MySQL Cluster 7.0 instead of MySQL Cluster 7.1 . This will give you the binaries and configurations you need.

Hi,

Can you upload schema + data and config.ini ?

/Jonas

I have encountered this bug on 7.1.14 on RHEL 5.6 while importing data from mysqldump. Reverting back to 7.1.13 fixed the issue.

My setup was 2 api nodes, 4 data nodes (ndbmtd, 2 node groups).

While running mysqldump with --extended-insert, the entire cluster would shutdown after approximately the same number of inserts regardless of dataset. Here is a sample of the cluster log file:

2011-08-10 12:21:05 [MgmtSrvr] ALERT    -- Node 3: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2011-08-10 12:21:05 [MgmtSrvr] ALERT    -- Node 1: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2011-08-10 12:21:05 [MgmtSrvr] ALERT    -- Node 49: Node 3 Disconnected
2011-08-10 12:21:05 [MgmtSrvr] ALERT    -- Node 49: Node 1 Disconnected
2011-08-10 12:21:05 [MgmtSrvr] ALERT    -- Node 49: Node 2 Disconnected
2011-08-10 12:21:06 [MgmtSrvr] INFO     -- Mgmt server state: nodeid 3 freed, m_reserved_nodes 1, 2, 4, 49 and 51.
2011-08-10 12:21:07 [MgmtSrvr] INFO     -- Mgmt server state: nodeid 1 freed, m_reserved_nodes 2, 4, 49 and 51.
2011-08-10 12:21:07 [MgmtSrvr] ALERT    -- Node 2: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2011-08-10 12:21:07 [MgmtSrvr] ALERT    -- Node 4: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2011-08-10 12:21:07 [MgmtSrvr] ALERT    -- Node 49: Node 4 Disconnected
2011-08-10 12:21:08 [MgmtSrvr] INFO     -- Mgmt server state: nodeid 2 freed, m_reserved_nodes 4, 49 and 51.
2011-08-10 12:21:09 [MgmtSrvr] INFO     -- Mgmt server state: nodeid 4 freed, m_reserved_nodes 49 and 51.

When I did mysqldump --skip-extended-insert, I still consistently reproduced node failure, but usually only one data node would fail and need to be restarted.

There is a mention in the changelog for 7.1.14 (Bug #61025) about fixing an issue with ndbmtd with a similar error message so this may possibly be related?

we found it,
and will release a new version asap

closing this as dup of #62116

/Jonas