Bug #33050 5.0.50 fails many SSL testcases

Description:
5.0.50 fails a lot of the SSL testcases.
openssl_1 rpl_openssl rpl_ssl ssl ssl_8k_key ssl_compress ssl_connect

See the attached log for the full failure output.
I can reproduce on x86_64, PPC64 and x86.

./configure --prefix=/usr --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libexecdir=/usr/sbin --sysconfdir=/etc/mysql --localstatedir=/var/lib/mysql --sharedstatedir=/usr/share/mysql --libdir=/usr/lib64/mysql --includedir=/usr/include/mysql --with-low-memory --with-client-ldflags=-lstdc++ --enable-thread-safe-client --with-comment=Gentoo Linux mysql-5.0.50 --without-docs --without-big-tables --enable-local-infile --with-extra-charsets=all --with-mysqld-user=mysql --with-server --with-unix-socket-path=/var/run/mysqld/mysqld.sock --without-libwrap --enable-shared --enable-static --without-debug --without-ndb-debug --with-charset=utf8 --with-collation=utf8_general_ci --with-embedded-privilege-control --with-embedded-server --with-bench --enable-assembler --with-extra-tools --with-innodb --without-readline --with-openssl --without-berkeley-db --with-geometry --with-ndbcluster --with-archive-storage-engine --with-csv-storage-engine --with-blackhole-storage-engine --with-federated-storage-engine --build=x86_64-pc-linux-gnu

System information:
Portage 2.1.4_rc4 (default-linux/amd64/2007.0, gcc-4.1.2, glibc-2.7-r0, 2.6.24-rc2-pmp-g28e80f62 x86_64)
=================================================================
System uname: 2.6.24-rc2-pmp-g28e80f62 x86_64 Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz
Timestamp of tree: Unknown
ccache version 2.4 [enabled]
app-shells/bash:     3.2_p17-r1
dev-lang/python:     2.5.1-r4
dev-python/pycrypto: 2.0.1-r6
dev-util/ccache:     2.4-r7
sys-apps/baselayout: 2.0.0_rc6
sys-apps/sandbox:    1.2.18.1-r2
sys-devel/autoconf:  2.13, 2.61-r1
sys-devel/automake:  1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r2, 1.10
sys-devel/binutils:  2.18-r1
sys-devel/gcc-config: 1.4.0-r4
sys-devel/libtool:   1.5.24
virtual/os-headers:  2.6.23-r2
dev-libs/openssl:    0.9.8g
ACCEPT_KEYWORDS="amd64 ~amd64"
CFLAGS="-march=nocona -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="-march=nocona -O2 -pipe"

How to repeat:
build + run testcases

Complete build and test output.

Attachment: 20071206-mysql-5.0.50-openssl-failures.txt.gz (application/gzip, text), 98.42 KiB.

5.0.51 also fails the same SSL testcases (plus one more, mysqlcheck).

I don't get the SSL failures on Debian, but the mysqlcheck failure as well:

--- /tmp/buildd/mysql-dfsg-5.0-5.0.51/mysql-test/r/mysqlcheck.result    2007-11-15 17:28:37.000000000 +0300
+++ /tmp/buildd/mysql-dfsg-5.0-5.0.51/mysql-test/var/log/mysqlcheck.reject      2007-12-07 12:56:29.000000000 +0300
@@ -49,7 +49,8 @@
 flush tables;
 removing and creating
 d_bug25347.t_bug25347
-error    : Incorrect file format 't_bug25347'
+Error    : Incorrect file format 't_bug25347'
+error    : Corrupt
 insert into t_bug25347 values (4),(5),(6);
 ERROR HY000: Incorrect file format 't_bug25347'
 d_bug25347.t_bug25347

mysqltest: Result content mismatch

Full build log of 5.0.51 on Debian 4.0/amd64.

Attachment: buildd.log.gz (application/x-gzip, text), 104.91 KiB.

Norbert: your Debian build log shows that you had SSL disabled via --without-openssl. Could you please set it up to build with openssl, and also mention what version of OpenSSL is on your system?

Indeed, the tests also fail with OpenSSL 0.9.8g on Debian when using --with-openssl.

The mysqlcheck test doesn't fail in 5.0.52.

SSL tests still fail with 5.0.52.

SSL tests fail for self-compiled 5.0.51 on Red Hat ES 4.6 (other tests pass except for "mysqlcheck" - see bug #33104.)

E.g. testing "openssl_1" works up and to the connection. It is likely that the behaviour of the client is not as expected in this environment:

---------- This gets executed ok: -----------

drop table if exists t1;
create table t1(f1 int);
insert into t1 values (5);
grant select on test.* to ssl_user1@localhost require SSL;
grant select on test.* to ssl_user2@localhost require cipher "DHE-RSA-AES256-SHA";
grant select on test.* to ssl_user3@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB/emailAddress=abstract.mys
ql.developer@mysql.com";
grant select on test.* to ssl_user4@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB/emailAddress=abstract.mys
ql.developer@mysql.com" ISSUER "/C=SE/ST=Uppsala/L=Uppsala/O=MySQL AB";
grant select on test.* to ssl_user5@localhost require cipher "DHE-RSA-AES256-SHA" AND SUBJECT "xxx";
flush privileges;

---------- The next instruction is: -----------

connect(localhost,ssl_user5,,test,MASTER_PORT,MASTER_SOCKET);

---------- With expected output: -----------

ERROR 28000: Access denied for user 'ssl_user5'@'localhost' (using password: NO)

---------- But what actually comes out is: -----------

mysqltest: At line 19: query 'connect  con1,localhost,ssl_user1,,,,,SSL' failed: 2026: SSL connection error

Thank you for the bug report. Verified on FC 6.0:

[miguel@amanhecer mysql-test]$ ./mysql-test-run.pl openssl_1
Logging: ./mysql-test-run.pl openssl_1
071220 20:55:42 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
071220 20:55:42 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
MySQL Version 5.0.56
Skipping ndbcluster, mysqld not compiled with ndbcluster
Setting mysqld to support SSL connections
Using MTR_BUILD_THREAD      = 0
Using MASTER_MYPORT         = 9306
Using MASTER_MYPORT1        = 9307
Using SLAVE_MYPORT          = 9308
Using SLAVE_MYPORT1         = 9309
Using SLAVE_MYPORT2         = 9310
Using IM_PORT               = 9312
Using IM_MYSQLD1_PORT       = 9313
Using IM_MYSQLD2_PORT       = 9314
Killing Possible Leftover Processes
Removing Stale Files
Creating Directories
Installing Master Database
=======================================================
Starting Tests in the 'main' suite

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

openssl_1                      [ fail ]

Configuring with --with-debug and running the test case like

  % ./mysql-test-run.pl --debug openssl_1

will show in "var/log/mysqltest.trace"

  error: SSL_do_handshake failure
  error: OpenSSL: error:140B4090:SSL routines:SSL_do_handshake:connection type not set:ssl_lib.c:2034:

As it turns out, OpenSSL contrary to yaSSL can't guess in
SSL_do_handshake() if called by a client or a server, you
need to call one of SSL_set_connect_state() or 
SSL_set_accept_state() before calling SSL_do_handshake().

I have a slight preference for Magnus' version of the patch, which passes a pointer to SSL_accept or SSL_connect to the helper function, instead of passing a flag.

It's frustrating that the api has this SSL_set_{connect,accept}_state() call, which isn't orthoganal with the separate SSL_accept and SSL_connect calls.  The whole SSL_do_handshake() thing is less user-friendly that it first appears.  Avoiding it makes the code clearer, I think.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/41851

ChangeSet@1.2573, 2008-02-07 08:48:28+01:00, msvensson@pilot.mysql.com +1 -0
  Bug#33050 5.0.50 fails many SSL testcases

Pushed into 5.1.24-rc

Pushed into 5.0.58

Pushed into 6.0.5-alpha

When MySQL was build with OpenSSL the SSL library was not properly initialized with information of which endpoint(server or client) it was, this failing to connect.

Noted in 5.0.58, 5.1.24, 6.0.5 changelogs.

Pushed into 5.1-telco-6.3.

Also documented in the 5.1.23-ndb-6.3.11 changelog.

Bug #40141 was marked as a duplicate of this one.