Bug #73030 Cluster goes down with Error 2341

Submitted: 18 Jun 2014 8:23 Modified: 17 Nov 2014 8:23
Reporter: erkan yanar Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql-5.6.15 ndb-7.3.4 OS:Linux
Assigned to: CPU Architecture:Any

[18 Jun 2014 8:23] erkan yanar
Description:
We have a cluster failing.
From the ndbd error log:

Time: Monday 19 May 2014 - 06:04:46
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: DbtupTrigger.cpp
Error object: DBTUP (Line: 2184) 0x00000002
Program: ndbd
Pid: 35713
Version: mysql-5.6.15 ndb-7.3.4
Trace: /usr/local/mysql/data/ndb_3_trace.log.8 [t1..t1]

As the second ndbd fails with the same error

Looking for the trace log I ended looking into  DbtuxMaint.cpp
133     treeAdd(c_ctx, frag, treePos, ent);
134     frag.m_entryCount++;
135     frag.m_entryBytes += searchKey.get_data_len();
136     frag.m_entryOps++;
137     break;
138   case TuxMaintReq::OpRemove:
139     jam();                                                                                                                                                           
140     ok = searchToRemove(c_ctx, frag, searchKey, ent, treePos);
141 #ifdef VM_TRACE
142     if (debugFlags & DebugMaint) {
143       debugOut << treePos << (! ok ? " - error" : "") << endl;
144     }
145 #endif
146     if (! ok) {
147       jam();
148       // there is no "Building" state so this will have to do
149       if (indexPtr.p->m_state == Index::Online) {
150         jam();
151         req->errorCode = TuxMaintReq::SearchError;
152       }
153       break;
154     }

So there is another state than online.
I.e. Building, Defining, Not Defined etc.

But this is not helping in any way. (at least for me)

How to repeat:
As we even don't know what triggers the shutdown, there is no way to find out how to reproduce it.
[18 Jun 2014 8:24] erkan yanar
Errorlog of node3

Attachment: ndb_3_error.log (text/x-log), 3.97 KiB.

[18 Jun 2014 8:25] erkan yanar
compressed log

Attachment: ndb_3_out.log.gz (application/gzip, text), 294.60 KiB.

[18 Jun 2014 8:26] erkan yanar
Trace log

Attachment: ndb_3_trace.log.8 (application/octet-stream, text), 1015.03 KiB.

[18 Jun 2014 8:26] erkan yanar
mgmd log

Attachment: ndb_1_cluster.log (text/x-log), 952.34 KiB.

[17 Nov 2014 8:20] MySQL Verification Team
Thank you for the report.
Colleague of mine confirmed that this is duplicate of Bug #56929

Thanks,
Umesh

Portions of this website are copyright © 2001, 2002 The PHP Group

Page generated in 0.041 sec. using MySQL 8.4.7-u6-cloud

Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the respective copyright holders.
It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.