Bug #49207 Nodefailure can "hang" ndb_mgmd with node id allocation
Submitted: 30 Nov 2009 13:59 Modified: 9 Dec 2009 12:56
Reporter: Jonas Oreland Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:mysql-5.1-telco-6.3 OS:Any
Assigned to: Jonas Oreland CPU Architecture:Any

[30 Nov 2009 13:59] Jonas Oreland
Description:
When api/mysqld/ndbd node is starting
it will allocate a node id, by sending a request to alive
cluster.

If the node that this request is sent to dies, the ndb_mgmd will
1) wait unnecessarily long, it will wait until node-failure handling is
   complete.
2) it unnecessarily will keep a mutex while waiting
3) other ndb_mgmd sessions, will unnecessarily block during disconnect
   due to the mutex

How to repeat:
1) have lots of clients connecting/disconnecting
e.g
bash> for i in `seq 25`; do eval "ndb_connect &"; done

2) have high show pressure
e.g 
bash> while true; do time ( ndb_mgm -e show > /dev/null ); done

3) have big DB with LCP running constantly (ndb_mgm -e "all dump 7017")

4) crash master in alloc node id (ndb_mgm -e "all error 934")

Observe time for show getting really big.

Suggested fix:
1) Stop waiting when NODE_FAILREP arrives (instead of NF_COMPLETEREP)
2) set reserved bit and release mutex prior to sending to kernel
   and clear it if allocation failed
3) don't block sessions if node-id is not allocated in the session
[30 Nov 2009 14:29] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/92098

3175 Jonas Oreland	2009-11-30
      ndb - bug#49207 - improve mutex handling wrt to alloc_node_id
[30 Nov 2009 14:52] Jonas Oreland
pushed to 6.3.29 and 7.0.10
[30 Nov 2009 14:52] Jonas Oreland
note to magnus: preferably this would have resulted in a new testcase in testMgmd, but I didn't...sorry
[1 Dec 2009 13:02] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/92273

3167 Martin Skold	2009-12-01 [merge]
      Merge
      modified:
        storage/ndb/src/common/debugger/EventLogger.cpp
        storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
        storage/ndb/src/kernel/blocks/ndbfs/AsyncIoThread.hpp
        storage/ndb/src/kernel/blocks/ndbfs/MemoryChannel.hpp
        storage/ndb/src/kernel/blocks/pgman.cpp
        storage/ndb/src/kernel/blocks/pgman.hpp
        storage/ndb/src/mgmsrv/MgmtSrvr.cpp
        storage/ndb/src/ndbapi/NdbOperationDefine.cpp
        storage/ndb/src/ndbapi/NdbOperationSearch.cpp
        storage/ndb/test/ndbapi/testBlobs.cpp
        storage/ndb/test/run-test/daily-basic-tests.txt
        storage/ndb/test/run-test/daily-devel-tests.txt
[1 Dec 2009 13:33] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/92279

3244 Martin Skold	2009-12-01 [merge]
      Merge
      modified:
        storage/ndb/src/common/debugger/EventLogger.cpp
        storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
        storage/ndb/src/kernel/blocks/ndbfs/AsyncIoThread.hpp
        storage/ndb/src/kernel/blocks/ndbfs/MemoryChannel.hpp
        storage/ndb/src/kernel/blocks/pgman.cpp
        storage/ndb/src/kernel/blocks/pgman.hpp
        storage/ndb/src/mgmsrv/MgmtSrvr.cpp
        storage/ndb/src/ndbapi/NdbOperationDefine.cpp
        storage/ndb/src/ndbapi/NdbOperationSearch.cpp
        storage/ndb/test/ndbapi/testBlobs.cpp
        storage/ndb/test/run-test/daily-basic-tests.txt
        storage/ndb/test/run-test/daily-devel-tests.txt
[1 Dec 2009 14:02] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/92287

3170 Martin Skold	2009-12-01 [merge]
      Merge
      modified:
        storage/ndb/src/common/debugger/EventLogger.cpp
        storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
        storage/ndb/src/kernel/blocks/ndbfs/AsyncFile.hpp
        storage/ndb/src/kernel/blocks/ndbfs/MemoryChannel.hpp
        storage/ndb/src/kernel/blocks/pgman.cpp
        storage/ndb/src/kernel/blocks/pgman.hpp
        storage/ndb/src/mgmsrv/MgmtSrvr.cpp
        storage/ndb/src/ndbapi/NdbOperationDefine.cpp
        storage/ndb/src/ndbapi/NdbOperationSearch.cpp
        storage/ndb/test/ndbapi/testBlobs.cpp
        storage/ndb/test/run-test/daily-basic-tests.txt
        storage/ndb/test/run-test/daily-devel-tests.txt
[9 Dec 2009 12:56] Jon Stephens
Documented bugfix in the NDB-6.3.29 and 7.0.10 changelogs as follows:

        If the master data node receiving a request from a newly-started
        API or data node for a node ID died before the request has been
        handled, the management server waited (and kept a mutex) until
        all handling of this node failure was complete before responding
        to any other connections, instead of responding to other
        connections as soon as it was informed of the node failure (that
        is, it waited until it had received a NF_COMPLETEREP signal
        rather than a NODE_FAILREP signal). On visible effect of this
        misbehavior was that it caused management client commands such
        as SHOW and ALL STATUS to respond with unnecessary slowness in
        such circumstances.

Closed.