MySQL Bugs: #29753: SSL Key Size Limit

Bug #29753	SSL Key Size Limit
Submitted:	12 Jul 2007 8:49	Modified:	9 Jul 2019 17:20
Reporter:	Gordan Bobic	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: General	Severity:	S3 (Non-critical)
Version:	5.1-bk, 5.0-bk, (5.1.20)	OS:	Any (RedHat 9, MacOSX, ..)
Assigned to:		CPU Architecture:	Any
Tags:	icc, openssl, rsa, SSL, yassl

Description:
It would appear that MySQL cannot handle 16384-bit RSA keys. When using 1024-bit CA/server/client keys, everything works correctly. When a 16384-bit key is used, the connection fails after a little while with the opaque:

ERROR 2026 (HY000): SSL connection error

At the very least the key size limit should be documented until this is fixed.

MySQL 5.1.20 was compiled from a source RPM. Spec file was doctored slightly to exclude things I don't use, namely NDB and InnoDB, and to make SSL work (yaSSL fails to compile using ICC due to a dirent vs. dirent64 pointer issue).

So, to summarize:
Compiler: ICC 9.1.051
OpenSSL: 0.9.7a-40 (RedHat 9)
Options:
CC=icc
CPP='icpc -E'
CXX=icpc
CPPFLAGS='-I/usr/kerberos/include'
--with-ssl=/usr/include

How to repeat:
Modify source RPM spec file to remove NDB and InnoDB (NDB at least seems to not want compile with ICC, but that's a different issue). Change the ssl option to --with-ssl=/usr/include

Cimpile with ICC:
export CC=icc CPP='icpc -E' CXX=icpc CPPFLAGS='-I/usr/kerberos/include'

Generate 16384-bit RSA CA/server/client keys.

Try to connect.

8192-bit keys don't work, either.

4096-bit keys DO work.

Upon further investigation, it appears that, additionally, remote SSL connections don't appear to work (2026 error) until the first local SSL connection is established. Afterwards they seem to work again. This is with 4096-bit keys.

Could it be that there is a time-out somewhere that trips with large keys (network lag + processing lag)?

Already filed Bug#29784 about similar issue in YaSSL. 
Anyway, OpenSSL support is slightly more interesting.

If YaSSL client connects without client-certificate, OpenSSL immediately rejects connection:
T@3    : | error: SSL_accept failure
T@3    : | >report_errors
T@3    : | | error: OpenSSL: error:140890E9:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:tls peer did not respond with certificate list:s3_srvr.c:1930:

OpenSSL client connects properly.

This seems to be another  bug, filed it as Bug#29841

now, if client key is provided (8k key too):

T@6    : | <vio_init
T@6    : | info: ssl: 0x310b9c0  timeout: 5
T@6    : | >vio_verify_callback
T@6    : | | enter: ok: 1  ctx: 0xb009262c
T@6    : | | info: cert: /C=SE/ST=Uppsala/O=MySQL AB/OU=Test/CN=CA
T@6    : | | exit: 1
T@6    : | <vio_verify_callback
T@6    : | >vio_verify_callback
T@6    : | | enter: ok: 1  ctx: 0xb009262c
T@6    : | | info: cert: /C=SE/ST=Uppsala/O=MySQL   \x0A     AB/OU=Test/CN=client
T@6    : | | exit: 1
T@6    : | <vio_verify_callback
T@6    : | error: SSL_accept failure
T@6    : | >report_errors
T@6    : | | error: OpenSSL: error:1408E098:SSL routines:SSL3_GET_MESSAGE:excessive message size:s3_both.c:444:

T@6    : | | error: error: error:00000005:lib(0):func(0):DH lib
T@6    : | | info: socket_errno: 0
T@6    : | <report_errors

Connecting with 2k client keys works properly. So it is client-key related, not server-key related. 

Verified against:
ChangeSet@1.2526, 2007-07-16 12:42:12+02:00, dfischer@mysql +1 -0
ChangeSet@1.2550, 2007-07-16 12:39:00+02:00, df@pippilotta +1 -0

With SSL libraries:
OpenSSL 0.9.8a 11 Oct 2005
OpenSSL 0.9.7l 28 Sep 2006

Just a side comment, not related to bug verification, such huge keys are usually outside scope of practical application.

Reverified with:
OpenSSL 0.9.8e 23 Feb 2007

Is this still an issue in MySQL 8.0? I guess 8.0.11 might have fixed this?

Have you considered trying it rather than asking?
The issue was trivially easy to reproduce.

Posted by developer:
 
We have migrated to modern SSL libraries that do support larger keys. If it still doesn't work for you please re-open the bug.