MySQL Bugs: #21611: Slave can't connect when master-ssl-cipher specified

Bug #21611	Slave can't connect when master-ssl-cipher specified
Submitted:	13 Aug 2006 11:13	Modified:	4 May 2007 18:24
Reporter:	Dan Lukes	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	5.0.37	OS:	FreeBSD (FreeBSD 6.2)
Assigned to:	Magnus Blåudd	CPU Architecture:	Any
Tags:	replication, SSL

Description:
Two mysqld servers with configured replication. The slave is configured to use SSL for connect, the master doesn't allow non-ssl connects from it.
The slave doesn't use the SSL, so master rejects the connection. Slave remaining within 'connecting to ...' state.

There is relevant part of debug trace:

--------------------------
...
| | | info: mysql protocol version 10, server=10
...
| | | >new_VioSSLFd
| | | | >my_malloc
| | | | | my: size: 4  my_flags: 0
| | | | | exit: ptr: 0x9bdb0d0
| | | | <my_malloc
| | | | error: failed to set ciphers to use
| | | | >report_errors
| | | | <report_errors
| | | | >my_free
| | | | | my: ptr: 0x9bdb0d0
| | | | <my_free
| | | <new_VioSSLFd
| | | >set_mysql_error
| | | | enter: error :2026 'SSL connection error'
| | | <set_mysql_error
| | | error: message: 2026/HY000 (SSL connection error)
| | | >end_server
...
| | >sql_print_error
| | | >vprint_msg_to_log
| | | | >print_buffer_to_file
| | | | | enter: buffer: Slave I/O thread: error connecting to master 'dzungle@(--hidden--):3306': Error: 'SSL connection error'  errno: 2026  retry-time: 60  retries: 86400
| | | | <print_buffer_to_file
| | | <vprint_msg_to_log
| | <sql_print_error
| | >thr_alarm
| | | enter: thread: T@3  sec: 120
| | | info: reschedule
| | <thr_alarm
...

--------------------------

As you can see, the failing function is SSL_CTX_set_cipher_list() called from vio/viosslfactories.c:new_VioSSLFd(). As SSL initialization failed, the code continue to establishing connection without SSL, but it's later denied by master server configuration.

The version of OpenSSL is 0.9.8b

MySQL 5.0.22 within the same environment, configuration and OpenSSL library version has no problem.

How to repeat:
Configure replication between two servers, force use of SSL on master.

Suggested fix:
As I used -O3 optimization during original compilation, I recompiled it with plain -O

I verified, that value of 'cipher' variable within new_VioSSLFd() is valid (I used the "ALL"). The only valid reason for fail of SSL_CTX_set_cipher_list() is "no required cipher avaiable", so the idea has been the underlying OpenSSL knew no cipher - e.g. it has not been initialized. I tried to add unconditional call of SSL_library_init() to begin of new_VioSSLFd() to be sure it IS initialized - but no change.  

I have no further idea about the reason, so I have no idea how to fix ...

Thank you for the report.

Could you please provide part of my.cnf corresponding to replication and ssl for master and for slave?

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

On master:

[mysqld]
max_connections=30
join_buffer_size = 524288
key_buffer_size = 64M
max_write_lock_count = 1024
record_buffer = 524288
query_cache_size = 12M
sort_buffer = 8M
default-character-set=utf8
log-bin=is-bin
set-variable = long_query_time=45
log-slow-queries
log-long-format
ssl
ssl-ca=cacert.pem
ssl-capath=/etc/ssl/msvpnCA/
ssl-cert=/etc/ssl/msvpnCA/is.crt
ssl-cipher=ALL
ssl-key=/etc/ssl/msvpnCA/is.key
server-id=1
skip-bdb
innodb_buffer_pool_size=80M;
new

On slave:
[mysqld]
join_buffer_size = 524288
key_buffer_size = 64M
max_write_lock_count = 1024
record_buffer = 524288
sort_buffer = 8M
default-character-set=utf8
log-bin=dzungle-bin
set-variable = long_query_time=45
log-long-format
skip-networking
server-id=50
master-host=is.******
master-port=3306
master-user= ******
master-password= ******
master-connect-retry=60
report-host=is-slave
replicate-do-db=radius_ms
master-ssl
master-ssl-ca=cacert.pem
master-ssl-capath=/etc/ssl/msvpnCA/
master-ssl-cert=/etc/ssl/msvpnCA/dzungle.crt
master-ssl-key=/etc/ssl/msvpnCA/dzungle.key
relay-log=dzungle-relay-bin
skip-bdb
new

By the way, in the meantime - OS has been updated to FreeBSD 6.2 with OpenSSL 0.9.8e

The problem has been retested on 5.0.37 and something changes - now I't possible to upgrade the slave to newest version - 5.0.37 slave works against 5.0.22 master. But it's forbidden to update master. It need to remain on 5.0.22

I need to correct previous statement. The information "something changes" is not true. The testing environment has been misconfigured. In the fact - nothing changes between 5.0.24 - 5.0.37 versions.

Yes, the call to 'SSL_CTX_set_cipher_list' fails since there is no cipher named ALL, at least not in yaSSL - the SSL library MySQL uses by default.

If you want to allow any cipher to be used just remove the "cipher=ALL" line
rom your my.cnf

The server is compiled and linked against OpenSSL, no yaSSL:
ldd /usr/local/libexec/mysqld 
/usr/local/libexec/mysqld:
...
        libcrypto.so.5 => /usr/local/lib/libcrypto.so.5 (0x284fe000)
...

And OpenSSL has ALL as a magic constant for SSL_CTX_set_cipher_list meaning "all but NULL ciphers". Also, SSL connections from regular clients (not SLAVEs) work with no problem.

Despite of it - BINGO! Commenting out the line solve the problem. Thank you.

Unfortunately, it confirm a bug exist, even it's another bug:

It seems this part of code use internal yaSSL despite of options during compilation.

Well, this bug is not critical.

If it's not bug but feature, then there shall be separate section in mysqld - one for replication connection served by yaSSL and second for casual connections which may be served by OpenSSL (if requested during compilation). As we can see, both library may require different configuration which is incompatible.

Also, there is feature request (althought it's also critical bug with impact to security) - if user select unsupported set of ciphers, then server should not start at all printing appropriate message to error log. User need to reevaluate it's decision. It's about security, isn't it ? Automagical changing of approved cipher set or, even worse, reverting to non-SSL connection (!!) is unacceptable behavior ...

Back to the trees - I made the same mistake as before. I tried to verify from slave who is not restricted to SSL connection so the replication work - but without SSL. So, forged the previous post. Forged about BINGO.

To avoid further misunderstanding, the recapitulation. The servers are compiled and linked against OpenSSL. Configuration files as previously attached, with ssl-cipher commented out. Master can be any version. For successful replication over SSL, the SLAVE must be 5.0.22 or older. Note, the SSL connection with regular  client work with no problem - only replication related connections has problems.

OK, sorry about confusion.

You say that running replication over SSL does not work although you have commented out the cipher=ALL line?

Please double check that the CA cert is the same on both master and slave and that the cert used on slave is signed by that same cert. When the slave connect it's cert will be verified against the CA cert of the master.

Yes. I advance, yesterday I miss another line - master-ssl-cipher on slave. Now I commented it also - no change.

As problem seems to be "master independent" I'm creating new slave's debugging environment now - debugging version of MySQL linked against libraries with debug informations (especially OpenSSL) on OS compiled with debug informations. Then I place breakpoint to problematic function and look what exactly happen within it, within SSL_CTX_new and SSL_CTX_set_cipher...

The keys should not be problem - the 5.0.22 work without problem using the same keys.

The difference tih 5.0.22 and the current version is that SSL_VERIFY_PEER is turned on by default.

/************************ VioSSLConnectorFd **********************************/
struct st_VioSSLFd *
new_VioSSLConnectorFd(const char *key_file, const char *cert_file,
                      const char *ca_file, const char *ca_path,
                      const char *cipher)
{
  struct st_VioSSLFd *ssl_fd;
  int verify= SSL_VERIFY_PEER;
              ^^
Here we tell the SSL library to verify that the cert of the other end - in the slave, we check the masters cert against the CA certs the slave. I suspsect that is where the problem occurs.

You can try by changing the above line to 
int verify= SSL_VERIFY_NONE;

Alse see BUG#25309 for a patch that makes it possible to run without ssl-ca and ssl-ca-path. Wich I thikn could be ueful in your case.

But of course the error message is very bad. That'll be worth a bug report in any case.

Damn!

At the first:
 --- viosslfactories.c:252 -------
/* Set the ciphers that can be used */
if (cipher && SSL_CTX_set_cipher_list(ssl_fd->ssl_context, cipher))
{
  DBUG_PRINT("error", ("failed to set ciphers to use"));
 ----------------------------------

At the second:
 --- from SSL_CTX_set_cipher_list() manual page --
RETURN VALUES
       SSL_CTX_set_cipher_list() return 1 if any cipher could be selected
       and 0 on complete failure.
 ----------------------------------

So - we have defined valid cipher so cipher is not NULL. The SSL_CTX_set_cipher_list() is called and return 1 as sign of "success" - at this time, the 'if (cipher && SSL_CTX_set_cipher_list(...))' enter the error handler ...

The yaSSL return SSL_SUCCES (=1). It seems the code doesn't work even with yaSSL.

We need to #define SSL_SUCCESS 1 when linking against OpenSSL then  

if (cipher && SSL_CTX_set_cipher_list(ssl_fd->ssl_context, cipher) != SSL_SUCCESS )

By the way, it's nice I commented out the cipher definition in my.cnf - but it's value is taken from master.info, so change of my.cnf changes nothing ...

Workaround - don't use any cipher definition, don't forget to remove it from master.info also.

As esential operations is disrupted (restriction of cipher set may be necesarry because of security) but workaround exist (at the cost of insufficient security) the bug severity is changed.

Also note, the original (5.0.24) version fallbacked to non-SSL version upon error which need to be classified as security bug. Current version (5.0.37) refuse to connect.

Synopsis and version updated accordingly.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/23530

ChangeSet@1.2421, 2007-04-02 13:12:59+02:00, msvensson@pilot.blaudden +3 -0
  Bug#21611 Slave can't connect when master-ssl-cipher specified
   - Change check for return value of 'SSL_CTX_set_cipher_list'
     in order to handle 0 as error setting cipher.
   - Thanks to Dan Lukes for finding the problem!

Pushed into 5.1.18-beta

Pushed into 5.0.42

Noted in 5.0.42, 5.1.18 changelogs.

A slave that used --master-ssl-cipher could not connect to the
master.