MySQL Bugs: #9270: multiple SSL race conditions

Bug #9270	multiple SSL race conditions
Submitted:	18 Mar 2005 10:39	Modified:	13 Oct 2005 2:34
Reporter:	Sergei Golubchik	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	4.0	OS:
Assigned to:	Sergei Glukhov	CPU Architecture:	Any

Description:
From: Leandro Santi
To: internals@lists.mysql.com
Subject: PATCH: multiple SSL race conditions.
Date: Wed, 23 Feb 2005 16:01:03 -0300
List-Archive: http://lists.mysql.com/internals/22381

The following patch,

http://webs.sinectis.com.ar/lesanti/misc/mysql-4.0.23a-openssl_locking.patch

Fixes a number of SSL race conditions on the MySQL server daemon. The
problem is subtle because MySQL keeps a dedicated SSL state object for
each client connection, so at first glance it would seem that no
inter-thread locking would be needed.

But, OpenSSL *needs* to be told about locking primitives, as documented:

http://www.openssl.org/docs/crypto/threads.html

I've found this problem while working on a MySQL-powered authentication
backend running on Solaris. The system ran fine for some hours, but after
some time new SSL connections are refused with an opaque ``ERROR 2026''
symptom. The server must be restarted in order to accept new conns.

You can easily reproduce the problem on a fresh source install of the 
database. Just follow this example:

% /etc/init.d/mysqld.sunos5 start
mysql service starting . ok  

% for j in 1 2 3 4 5 6 7 8 9; do while [ 1 ]; do echo quit | /usr/local/mysql/bin/mysql -ussluser --ssl-key=/usr/local/mysql/ssl/client-key.pem test && echo ok.$j; done & done
[1] 4987
[2] 4989
ERROR 2026: SSL connection error
[3] 4992
ok.2
[4] 4997
[5] 5002
ok.3
ok.2
[6] 5005
ok.1
ok.4
ERROR 2026: SSL connection error
ok.5
[7] 5012
[8] 5022
ok.6
ok.1
ok.4
ERROR 2026: SSL connection error
ok.4
ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
[9] 5026
[root@lazy1 (impsat) mysql-4.0.23a]$ ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
ok.3
ok.2
...

After a while, the interal data structures get totally corrupted, and
the system refuses every new SSL connection:

ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
ERROR 2026: SSL connection error
...

The problem seems to be related to the handling of the RSA blinding
objects inside the SSL library. An instance of this object is being
shared between multiple SSL contexts, so MySQL really has to pass the
required locking primitives at startup.

Also, note that corruption can happen during the initial SSL handshaking
at the very first stages of the MySQL session, so anyone can trigger the
problem from the outside.

Bye, Leandro.

pd: Note that on Linux the problem is much more difficult to trigger,
because of its getpid() semantics. Please, take a look at the patch's
notes for further information.

How to repeat:

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/26444

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/30144

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/30901

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/30904

Fixed in 4.1.16

Documented in 4.1.16 changelog:

<listitem>
        <para>
          Multiple race conditions existed in SSL, particularly
          noticeable on Solaris. (Bug #9270)
        </para>
      </listitem>