Bug #65141 Data node crash in error 2341 in MySQL Cluster 7.2.5
Submitted: 28 Apr 2012 17:50 Modified: 17 May 2012 20:14
Reporter: Wei-bin Zhang Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:7.2.5 OS:Linux (CentOS 5)
Assigned to: Assigned Account CPU Architecture:Any
Tags: cluster, crash, data node, stability

[28 Apr 2012 17:50] Wei-bin Zhang
Description:
Repeatable all data node crash every several hours in production system.

How to repeat:
Start with cluster running -- with 2 mysql nodes, and 4 data nodes starting up successfully with  data already imported. 

Cluster looks running well at beginning -- generally lasting 2-6 hours.

Closter crash suddenly, all data nodes crashed.

The cluster crash twice in one day, all log list below:

Time: Saturday 28 April 2012 - 09:40:39
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbspjMain.cpp
Error object: DBSPJ (Line: 888) 0x00000002
Program: ndbd
Pid: 26468
Version: mysql-5.5.20 ndb-7.2.5
Trace: /data/mysql/data/ndb_2_trace.log.1 [t1..t1]
***EOM***
                                                                                       
Time: Saturday 28 April 2012 - 14:30:44
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DblqhMain.cpp
Error object: DBLQH (Line: 9785) 0x00000002
Program: ndbd
Pid: 4766
Version: mysql-5.5.20 ndb-7.2.5
Trace: /data/mysql/data/ndb_2_trace.log.2 [t1..t1]
***EOM***
[3 May 2012 17:32] Alexandr Yakunin
I have the same issue 
Version:	7.2.5
OS Gentoo

It happens when we try to use ndb_join_pushdown
[3 May 2012 17:34] Alexandr Yakunin
Time: Thursday 3 May 2012 - 19:01:49
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbspjMain.cpp
Error object: DBSPJ (Line: 888) 0x00000002
Program: ndbd
Pid: 21554
Version: mysql-5.5.20 ndb-7.2.5
Trace: /var/lib/mysql/ndb_data/ndb_11_trace.log.8 [t1..t1]
***EOM***
[8 May 2012 7:19] Ole John Aske
This could potentially be two different bugs:

1) Crash in DPSPJ, line 888 which is a duplicate of Bug#64865
2) Crash in DBLQH, line 9785 which is yet not reproduced / fixed.
   Thus it is unknown if this is a genuine bug, or caused by the first.
[17 May 2012 20:14] Ole John Aske
This other bug (Crash in DBLQH, Line: 9785) is a duplicate of Bug#65084:
'7.2.4 and 7.2.5 crash in DBLQH under mild load'. The line number is a little bit of, but that is due to this bug is reported on 7.2.4.

So I will close this bug report as a duplicate as we already have two other reports covering the same issues.