MySQL Bugs: #24148: regression tests hang with SSL enabled

Bug #24148	regression tests hang with SSL enabled
Submitted:	9 Nov 2006 18:46	Modified:	31 Jan 2007 19:31
Reporter:	[ name withheld ]	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	5.0.27	OS:	Linux (Fedora Core 5)
Assigned to:	Magnus Blåudd	CPU Architecture:	Any

Description:
Some of the mysql regression tests, notably the "kill" test, repeatably hang when 5.0.27 is built --with-openssl.  Debugging reveals that vio_ssl_close() has been modified recently to call SSL_shutdown twice, that is, it not only sends a shutdown notification to the client but waits for a response to come back.  But at this point (in the affected tests) mysqltest is busy doing something else and won't respond immediately.  This wouldn't be a big problem except that vio_ssl_close is called from close_connection while the latter is holding the LOCK_thread_count mutex.  That means that, for instance, no new SQL query can start.  Instant deadlock, because that's exactly what mysqltest is waiting for.

How to repeat:
Try to run regression tests in an OpenSSL build.

Suggested fix:
I should think that doing any network I/O whatsoever while holding this mutex is an abysmally bad idea.  Can't close_connection be reorganized so that neither net_send_error nor vio_close are done with the lock held?

Your observations about holding the mutex while performing network IO are very interesting. I will of course take a look at that.

Please post your patches to this bug report here.

Calling SSL_shutdown twice is per instructions from http://www.openssl.org/docs/ssl/SSL_shutdown.html - but as it also says the second  call is not required if "the underlying connection shall be closed anyway".

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/17286

ChangeSet@1.2352, 2006-12-22 00:05:40+01:00, msvensson@pilot.mysql.com +1 -0
  Bug#24148 regression tests hang with SSL enabled
   - Don't call SSL_shutdown a second time

This nasty bug is happening in real world usage scenarios. In my case:

set-variable = interactive_timeout=180
set-variable = wait_timeout=180

now we connect using SSL and commandline mysql client to mysql server and leave client idle. After 180s server starts disconnection of the client and it hangs there. Unfortunately the hang is nasty one - no one else is able to connect or issue any commands until SSL client disconnects.

I can also confirm that pending patch fixes that issue, too.

Thank you Arkadiusz.

Noted in 5.0.36, 5.1.15 changelogs.

SSL connections could hang at connection shutdown.