MySQL Bugs: #24148: regression tests hang with SSL enabled

Bug #24148	regression tests hang with SSL enabled
Submitted:	9 Nov 2006 19:46	Modified:	31 Jan 2007 20:31
Reporter:	[ name withheld ]
Status:	Closed
Category:	Server	Severity:	S3 (Non-critical)
Version:	5.0.27	OS:	Linux (Fedora Core 5)
Assigned to:	Magnus Svensson	Target Version:

Description:
Some of the mysql regression tests, notably the "kill" test, repeatably hang when 5.0.27
is built --with-openssl.  Debugging reveals that vio_ssl_close() has been modified
recently to call SSL_shutdown twice, that is, it not only sends a shutdown notification to
the client but waits for a response to come back.  But at this point (in the affected
tests) mysqltest is busy doing something else and won't respond immediately.  This
wouldn't be a big problem except that vio_ssl_close is called from close_connection while
the latter is holding the LOCK_thread_count mutex.  That means that, for instance, no new
SQL query can start.  Instant deadlock, because that's exactly what mysqltest is waiting
for.

How to repeat:
Try to run regression tests in an OpenSSL build.

Suggested fix:
I should think that doing any network I/O whatsoever while holding this mutex is an
abysmally bad idea.  Can't close_connection be reorganized so that neither net_send_error
nor vio_close are done with the lock held?

Your observations about holding the mutex while performing network IO are very
interesting. I will of course take a look at that.

Please post your patches to this bug report here.

Calling SSL_shutdown twice is per instructions from
http://www.openssl.org/docs/ssl/SSL_shutdown.html - but as it also says the second  call
is not required if "the underlying connection shall be closed anyway".

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/17286

ChangeSet@1.2352, 2006-12-22 00:05:40+01:00, msvensson@pilot.mysql.com +1 -0
  Bug#24148 regression tests hang with SSL enabled
   - Don't call SSL_shutdown a second time

This nasty bug is happening in real world usage scenarios. In my case:

set-variable = interactive_timeout=180
set-variable = wait_timeout=180

now we connect using SSL and commandline mysql client to mysql server and leave client
idle. After 180s server starts disconnection of the client and it hangs there.
Unfortunately the hang is nasty one - no one else is able to connect or issue any commands
until SSL client disconnects.

I can also confirm that pending patch fixes that issue, too.

Thank you Arkadiusz.

Noted in 5.0.36, 5.1.15 changelogs.

SSL connections could hang at connection shutdown.