Bug #37592 ndbd nodes crashed in DbtupPageMap.cpp
Submitted: 23 Jun 2008 17:38 Modified: 9 May 2009 12:59
Reporter: Serge Kozlov Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Replication Severity:S2 (Serious)
Version:mysql-5.1-telco-6.* OS:Any
Assigned to: CPU Architecture:Any

[23 Jun 2008 17:38] Serge Kozlov
Description:
Master ndbd nodes crashed during stress testing with following errors:

Current byte-offset of file-pointer is: 568

Time: Saturday 21 June 2008 - 01:23:56
Status: Temporary error, restart node
Message: Another node failed during system restart, please investigate error(s) on other node(s) (Restart error)
Error: 2308
Error data: Node 3 disconnected
Error object: QMGR (Line: 2766) 0x00000002
Program: /data0/cr_autotest/libexec/ndbd
Pid: 21057
Trace: ./ndb_2_trace.log.1
Version: mysql-5.1.24 ndb-6.3.16-RC
***EOM***
         
                                                                                  Current byte-offset of file-pointer is: 568

Time: Saturday 21 June 2008 - 01:23:55
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtup/DbtupPageMap.cpp
Error object: DBTUP (Line: 481) 0x00000002
Program: /data0/cr_autotest/libexec/ndbd
Pid: 21115
Trace: ./ndb_3_trace.log.1
Version: mysql-5.1.24 ndb-6.3.16-RC
***EOM***
                                                                        

How to repeat:
See attached log files.
[23 Jun 2008 17:42] Magnus Blåudd
mysql-5.1-telco-6.3
[23 Jun 2008 17:45] Magnus Blåudd
It hits the ndbrequire at line 481 in DbtupPageMap.cpp

// Out of memory ?? Should nto be possible here/now
ndbrequire(nextPtr != 0 && prevPtr != 0);
[23 Jun 2008 18:07] Magnus Blåudd
One can see from the signal log how the rebuild is reaching higher and higher "pageId" and "tail"

--------------- Signal ----------------
r.bn: 249 "DBTUP", r.proc: 3, r.sigId: 976942476 gsn: 164 "CONTINUEB" prio: 1
s.bn: 249 "DBTUP", s.proc: 3, s.sigId: 976942475 length: 4 trace: 1 #sec: 0 fragInf: 0
 H'0000000f H'00000000 H'3a20ba00 H'3a20b9ff
              ^ Nada     ^ pageId   ^ tail
   ^ ZREBUILD_FREE_PAGE_LIST
[9 Apr 2009 12:59] Jonathan Miller
We need information on how to repeat and maybe if still happens on 6.4 telco.
[9 May 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".