MySQL Bugs: #38563: ndb_mgmd performs network I/O with client(s) while holding mutex

Bug #38563	ndb_mgmd performs network I/O with client(s) while holding mutex
Submitted:	5 Aug 2008 9:50	Modified:	5 Oct 2008 16:29
Reporter:	Magnus Blåudd	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S3 (Non-critical)
Version:	6.2	OS:	Any
Assigned to:	Magnus Blåudd	CPU Architecture:	Any

Description:
The ndb_mgmd will perform network I/O with the client while holding the MgmApiSesion mutex and sometimes also the SocketServer::m_session_mutex. This in combination with not checking the return value when printing to m_outputstream leads to long running threads (that are writing to clients that no longer exist) holding mutexes in the ndb_mgmd.

How to repeat:
Run ./testMgm in a loop. Second loops will take really long time before the threads disconnected(and thus hanging) in the first loop will timeout and relase the mutex so that other threads may join.

Suggested fix:
1. Use BufferedOutputStream that will buffer the result and write it when the mutexes has been released.
2. Avoid locking the m_session_mutex by taking a copy of the session pointers and release the m_session_mutex while calling the callback function passed to 'SocketServer::foreachSession'

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/50918

2637 Magnus Svensson	2008-08-05
      Bug#38563 ndb_mgmd performs network I/O with client(s) while holding mutex

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/50919

2645 Magnus Svensson	2008-08-05 [merge]
      Merge bug#38563

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/50923

2697 Magnus Svensson	2008-08-05 [merge]
      Merge bug#38563

Documented bugfix in the NDB 6.2.16 and 6.3,17 changelogs as follows:

        ndb_mgmd performed unnecessary network I/O with the client. This in
        combination with other factors led to long-running threads that were
        attempting to write to clients that no longer existed.

Already documented for NDB 6.2/6.3; closed.

Pushed into 6.0.6-alpha  (revid:msvensson@mysql.com-20080805115352-uyhcspxinmbgp1kb) (version source revid:jonas@mysql.com-20080813092004-7zlf6eu87i4ziwm2) (pib:5)