Bug #11516 ndb_mgmd debug core on cluster shutdown with failed data nodes
Submitted: 22 Jun 2005 18:38 Modified: 20 Jul 2005 15:14
Reporter: Jonathan Miller Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:5.0.9, 5.1.0-wl2325-wl1354-new OS:Linux (Linux)
Assigned to: Stewart Smith CPU Architecture:Any

[22 Jun 2005 18:38] Jonathan Miller
Description:
I had a 4 data node cluster running on ndb08 and ndb09. I had left the BANK test running over night. ndb08 had ran out of disk space, so the data nodes on that system had shutdown. I had stopped the bank test around 8:AM this morning leaving the ndb_mgmd process running on ndb08 and both data nodes running on ndb09 while I was testing a mysqld patch from mats for 11366 on the slave system.

I returned after lunch to cleanup the systems and issues ndb_mgm ndb08:14000 -e "shutdown" to shutdown the cluster, after which I found a core. The trace shows as follows:
(Note: Not sure about the stop=0x0)

#0  0x00000018 in ?? ()
#1  0x0806e526 in Parser<MgmApiSession>::run(Parser<MgmApiSession>::Context&, MgmApiSession&, bool volatile*) const (this=0x84af710, ctx=@0xbd9ff7ec,
    t=@0x88c2958, stop=0x0) at Parser.hpp:177
#2  0x080699eb in MgmApiSession::runSession() (this=0x88c2958)
    at Services.cpp:298
#3  0x400d42cc in sessionThread_C (_sc=0x88c2958) at SocketServer.cpp:334
#4  0x400c9ca8 in ndb_thread_wrapper (_ss=0x8454de8) at NdbThread.c:58
#5  0x401276de in pthread_start_thread () from /lib/libpthread.so.0

(gdb) frame 1
#1  0x0806e526 in Parser<MgmApiSession>::run(Parser<MgmApiSession>::Context&, MgmApiSession&, bool volatile*) const (this=0x84af710, ctx=@0xbd9ff7ec,
    t=@0x88c2958, stop=0x0) at Parser.hpp:177
177     Parser.hpp: No such file or directory.
        in Parser.hpp
(gdb) list
172     in Parser.hpp

(gdb) frame 2
#2  0x080699eb in MgmApiSession::runSession() (this=0x88c2958)
    at Services.cpp:298
298         m_parser->run(ctx, *this);
(gdb) list
293
294     void
295     MgmApiSession::runSession() {
296       Parser_t::Context ctx;
297       while(!m_stop) {
298         m_parser->run(ctx, *this);
299
300         if(ctx.m_currentToken == 0)
301           break;
302
(gdb)

File will be moved to ndb08: /space/bug##### where #### is the number of this report.

How to repeat:
See above (not easy)
[29 Jun 2005 22:01] Jonathan Miller
Had it happen again in testing a fix from Mats on a different bug. Seems to be easy to reproduce.

Here is another back trace from a different core file:
 0x40245124 in strcmp () from /lib/libc.so.6
#1  0x400a318f in ParserImpl::matchCommand(Parser<ParserImpl::Dummy>::Context*, char const*, ParserRow<ParserImpl::Dummy> const*) ()
   from /home/ndbdev/jmiller/builds/lib/mysql/libndbclient.so.0
#2  0x400a2e3c in ParserImpl::run(Parser<ParserImpl::Dummy>::Context*, Properties const**, bool volatile*) const ()
   from /home/ndbdev/jmiller/builds/lib/mysql/libndbclient.so.0
#3  0x080683fe in Parser<MgmApiSession>::run(Parser<MgmApiSession>::Context&, MgmApiSession&, bool volatile*) const ()
#4  0x08064053 in MgmApiSession::runSession() ()
#5  0x400a45d0 in sessionThread_C ()
   from /home/ndbdev/jmiller/builds/lib/mysql/libndbclient.so.0
#6  0x4009aff1 in ndb_thread_wrapper ()
   from /home/ndbdev/jmiller/builds/lib/mysql/libndbclient.so.0
#7  0x400e86de in pthread_start_thread () from /lib/libpthread.so.0
[29 Jun 2005 22:03] Jonathan Miller
Note: the last core was not on a debug build, but rather a release build.
[6 Jul 2005 4:41] Stewart Smith
Duplicated in 5.0 bk.
[7 Jul 2005 5:25] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/26747
[12 Jul 2005 5:58] Stewart Smith
Pushed to 5.0.10
[20 Jul 2005 15:14] Jon Stephens
Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html

Additional info:

Documented bugfix for 5.0.10.