MySQL Bugs: #44590: Error: 2341 on NDB node Shuts down cluster

Bug #44590	Error: 2341 on NDB node Shuts down cluster
Submitted:	30 Apr 2009 21:54	Modified:	6 Jun 2009 9:16
Reporter:	Ari Cooperman	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	mysql-5.1-telco-7.0	OS:	Linux (RedHat ES 4)
Assigned to:		CPU Architecture:	Any
Tags:	mysql-5.1.32 ndb-7.0.5-beta

Description:
4 NDB (ndbmtd) node, 1 MDM and 1 mysqld cluster, doing batch inserts under decently heavy load from 10 clients. MGM console shows:

ndb_mgm> Node 4: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
Node 2: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'.
Node 3: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'.
Node 5: Forced node shutdown completed. Caused by error 2305: 'Node lost connection to other nodes and can not form a unpartitioned cluster, please investigate if there are error(s) on other node(s)(Arbitration error). Temporary error, restart node'.

The error log of the node that failed shows:

Version: mysql-5.1.32 nd
Time: Thursday 30 April 2009 - 17:15:35
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing erro
r message, please report a bug)
Error: 2341
Error data: SimulatedBlock.cpp
Error object: DBTC (Line: 501) 0x0000000a

Trace logs attached

How to repeat:
Start up same configuration and send high insert load via JDBC

Hi,

can you please upload error-logs/tracefiles/cluster-logs and config.ini
(possibly using ndb_error_reporter.pl)

Unfortunately the trace files are bigger than 500K and I can't access the ftp drop site from work. Is there another method that I can upload the files?

You can try mailing them to me (hartmut <at> mysql <dot> com) if that works for you, i'd then take care of transfering them to the FTP server and referencing the FTP URLs here ...

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".