Bug #24148 regression tests hang with SSL enabled
Submitted: 9 Nov 2006 18:46 Modified: 31 Jan 2007 19:31
Reporter: [ name withheld ] Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:5.0.27 OS:Linux (Fedora Core 5)
Assigned to: Magnus Blåudd CPU Architecture:Any

[9 Nov 2006 18:46] [ name withheld ]
Description:
Some of the mysql regression tests, notably the "kill" test, repeatably hang when 5.0.27 is built --with-openssl.  Debugging reveals that vio_ssl_close() has been modified recently to call SSL_shutdown twice, that is, it not only sends a shutdown notification to the client but waits for a response to come back.  But at this point (in the affected tests) mysqltest is busy doing something else and won't respond immediately.  This wouldn't be a big problem except that vio_ssl_close is called from close_connection while the latter is holding the LOCK_thread_count mutex.  That means that, for instance, no new SQL query can start.  Instant deadlock, because that's exactly what mysqltest is waiting for.

How to repeat:
Try to run regression tests in an OpenSSL build.

Suggested fix:
I should think that doing any network I/O whatsoever while holding this mutex is an abysmally bad idea.  Can't close_connection be reorganized so that neither net_send_error nor vio_close are done with the lock held?
[28 Nov 2006 9:02] Magnus Blåudd
Your observations about holding the mutex while performing network IO are very interesting. I will of course take a look at that.

Please post your patches to this bug report here.

Calling SSL_shutdown twice is per instructions from http://www.openssl.org/docs/ssl/SSL_shutdown.html - but as it also says the second  call is not required if "the underlying connection shall be closed anyway".
[21 Dec 2006 23:05] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/17286

ChangeSet@1.2352, 2006-12-22 00:05:40+01:00, msvensson@pilot.mysql.com +1 -0
  Bug#24148 regression tests hang with SSL enabled
   - Don't call SSL_shutdown a second time
[29 Dec 2006 15:39] Arkadiusz Miskiewicz
This nasty bug is happening in real world usage scenarios. In my case:

set-variable = interactive_timeout=180
set-variable = wait_timeout=180

now we connect using SSL and commandline mysql client to mysql server and leave client idle. After 180s server starts disconnection of the client and it hangs there. Unfortunately the hang is nasty one - no one else is able to connect or issue any commands until SSL client disconnects.

I can also confirm that pending patch fixes that issue, too.
[15 Jan 2007 12:04] Magnus Blåudd
Thank you Arkadiusz.
[31 Jan 2007 19:31] Paul DuBois
Noted in 5.0.36, 5.1.15 changelogs.

SSL connections could hang at connection shutdown.