MySQL Bugs: #69417: Shutdown MySQL Cluster Nodes error:2341

Bug #69417	Shutdown MySQL Cluster Nodes error:2341
Submitted:	6 Jun 2013 16:35	Modified:	19 Sep 2013 12:22
Reporter:	Rodolfo Mijangos	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	ndb-7.2.5	OS:	Other (Debian 6)
Assigned to:	Assigned Account	CPU Architecture:	Any
Tags:	forced node, restart node, Shutdown nodes

Description:
I have mysql 2 nodes mysql-5.5.20 with ndb-7.2.5.,  this nodes forced to shutdown. 

How to repeat:
This is my Cluster Configuration 

Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=3    @192.168.0.9  (mysql-5.5.20 ndb-7.2.5, Nodegroup: 0)
id=4    @192.168.0.10  (mysql-5.5.20 ndb-7.2.5, Nodegroup: 0, Master)

[ndb_mgmd(MGM)] 2 node(s)
id=1    @192.168.0.9  (mysql-5.5.20 ndb-7.2.5)
id=2    @192.168.0.10  (mysql-5.5.20 ndb-7.2.5)

[mysqld(API)]   2 node(s)
id=5    @192.168.0.9  (mysql-5.5.20 ndb-7.2.5)
id=6    @192.168.0.10  (mysql-5.5.20 ndb-7.2.5)

I have this logs 

2013-06-06 13:53:12 [MgmtSrvr] INFO     -- Node 4: Local checkpoint 9232 completed
2013-06-06 13:53:13 [MgmtSrvr] INFO     -- Node 4: Local checkpoint 9233 started. Keep GCI = 4021138 oldest restorable GCI = 4021429
2013-06-06 14:30:06 [MgmtSrvr] ALERT    -- Node 4: Node 3 Disconnected
2013-06-06 14:30:06 [MgmtSrvr] INFO     -- Node 4: Communication to Node 3 closed
2013-06-06 14:30:06 [MgmtSrvr] ALERT    -- Node 4: Network partitioning - arbitration required
2013-06-06 14:30:06 [MgmtSrvr] INFO     -- Node 4: President restarts arbitration thread [state=7]
2013-06-06 14:30:06 [MgmtSrvr] ALERT    -- Node 2: Node 3 Disconnected
2013-06-06 14:30:07 [MgmtSrvr] ALERT    -- Node 4: Arbitration won - positive reply from node 1
2013-06-06 14:30:07 [MgmtSrvr] ALERT    -- Node 3: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2013-06-06 14:30:07 [MgmtSrvr] INFO     -- Node 4: Started arbitrator node 1 [ticket=56b0000f31ac5a22]
2013-06-06 14:31:06 [MgmtSrvr] WARNING  -- Node 4: Failure handling of node 3 has not completed in 1 min - state = 6
2013-06-06 14:31:18 [MgmtSrvr] INFO     -- Node 4: Local checkpoint 9233 completed
2013-06-06 14:31:19 [MgmtSrvr] INFO     -- Node 4: Communication to Node 3 opened
2013-06-06 14:31:19 [MgmtSrvr] INFO     -- Node 4: Local checkpoint 9234 started. Keep GCI = 4021674 oldest restorable GCI = 4022427
2013-06-06 14:58:38 [MgmtSrvr] ALERT    -- Node 4: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2013-06-06 14:58:38 [MgmtSrvr] ALERT    -- Node 2: Node 4 Disconnected.    

I ran the ndb_error_reporter and show this message.

2013-06-06 16:10:40 [ndb_config] WARNING  -- at line 55: Cluster configuration warning:
  arbitrator with id 1 and db node with id 3 on same host dbsvr1.cluster
  arbitrator with id 2 and db node with id 4 on same host dbsvr2.cluster
  Running arbitrator on the same host as a database node may
  cause complete cluster shutdown in case of host failure.
2013-06-06 16:10:40 [ndb_config] WARNING  -- at line 55: Cluster configuration warning:
  arbitrator with id 1 and db node with id 3 on same host dbsvr1.cluster
  arbitrator with id 2 and db node with id 4 on same host dbsvr2.cluster
  Running arbitrator on the same host as a database node may
  cause complete cluster shutdown in case of host failure.

you have any solution?

To analyze this kind of error the data node error log, trace files and output log files are needed ... preferably collected using the ndb_error_reporter tool

Hi

The ndb_error_reporter already gave you a hint about what could be the reason:
-->
I ran the ndb_error_reporter and show this message.

2013-06-06 16:10:40 [ndb_config] WARNING  -- at line 55: Cluster configuration warning:
  arbitrator with id 1 and db node with id 3 on same host dbsvr1.cluster
  arbitrator with id 2 and db node with id 4 on same host dbsvr2.cluster
  Running arbitrator on the same host as a database node may
  cause complete cluster shutdown in case of host failure.
-->

To tell if this is the reason you would also need to attach the file generated by ndb_error_reporter. The report should be small enough to attach to the bug.

/Gustaf

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".