| Bug #42158 | leak: SSL_get_peer_certificate() doesn't have matching X509_free() | ||
|---|---|---|---|
| Submitted: | 16 Jan 2009 13:48 | Modified: | 25 Jun 2009 22:37 |
| Reporter: | Liam Kelleher | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: General | Severity: | S2 (Serious) |
| Version: | 4.1, 5.0, 5.1, 6.0 | OS: | Any |
| Assigned to: | Davi Arnaut | CPU Architecture: | Any |
| Tags: | Leak openssl | ||
[16 Jan 2009 13:48]
Liam Kelleher
[16 Jan 2009 19:28]
Sveta Smirnova
Thank you for the report. Please indicate accurate version numbers of servers do you run and send us valgrind output.
[19 Jan 2009 9:56]
Liam Kelleher
The versions I tests witha re 5.0.45 and 4.1.22. valgrind output: ==2578== LEAK SUMMARY: ==2578== definitely lost: 320,544 bytes in 2,220 blocks. ==2578== indirectly lost: 10,621,275 bytes in 254,219 blocks. ==2578== possibly lost: 37,655 bytes in 136 blocks. ==2578== still reachable: 138,472 bytes in 2,607 blocks. ==2578== suppressed: 0 bytes in 0 blocks. ==2578== Reachable blocks (those to which a pointer was found) are not shown. ==2578== To see them, rerun with: --leak-check=full --show-reachable=yes ==14885== 4,084 bytes in 20 blocks are possibly lost in loss record 3 of 9 ==14885== at 0x49050E7: malloc (vg_replace_malloc.c:207) ==14885== by 0x4BB5CBA: CRYPTO_malloc (mem.c:304) ==14885== by 0x4C00AA3: BUF_MEM_grow (buffer.c:110) ==14885== by 0x4C2EB72: X509_NAME_oneline (x509_obj.c:85) ==14885== by 0x4C1D751: x509_cb (x_x509.c:101) ==14885== by 0x4C23F10: ASN1_item_ex_d2i (tasn_dec.c:399) ==14885== by 0x4C24393: ASN1_item_d2i (tasn_dec.c:116) ==14885== by 0x4A3FD65: ssl3_get_client_certificate (s3_srvr.c:1971) ==14885== by 0x4A416F1: ssl3_accept (s3_srvr.c:438) ==14885== by 0x8847E4: sslaccept (viossl.c:206) ==14885== by 0x58D05A: handle_one_connection (in /usr/src/redhat/BUILD/mysql-5.0.45/sql/mysqld) ==14885== by 0x3964606109: start_thread (in /lib64/tls/libpthread-2.3.4.so) ==14885== ==14885== ==14885== 12,384 bytes in 43 blocks are possibly lost in loss record 5 of 9 ==14885== at 0x490695A: calloc (vg_replace_malloc.c:397) ==14885== by 0x3963B0D262: _dl_allocate_tls (in /lib64/ld-2.3.4.so) ==14885== by 0x3964606706: pthread_create@@GLIBC_2.2.5 (in /lib64/tls/libpthread-2.3.4.so) ==14885== by 0x56FBE4: main (in /usr/src/redhat/BUILD/mysql-5.0.45/sql/mysqld) ==14885== ==14885== ==14885== 8,033,141 (233,856 direct, 7,799,285 indirect) bytes in 1,624 blocks are definitely lost in loss record 9 of 9 ==14885== at 0x49050E7: malloc (vg_replace_malloc.c:207) ==14885== by 0x4BB5CBA: CRYPTO_malloc (mem.c:304) ==14885== by 0x4C21740: asn1_item_ex_combine_new (tasn_new.c:170) ==14885== by 0x4C24222: ASN1_item_ex_d2i (tasn_dec.c:318) ==14885== by 0x4C24393: ASN1_item_d2i (tasn_dec.c:116) ==14885== by 0x4A3FD65: ssl3_get_client_certificate (s3_srvr.c:1971) ==14885== by 0x4A416F1: ssl3_accept (s3_srvr.c:438) ==14885== by 0x8847E4: sslaccept (viossl.c:206) ==14885== by 0x58D05A: handle_one_connection (in /usr/src/redhat/BUILD/mysql-5.0.45/sql/mysqld) ==14885== by 0x3964606109: start_thread (in /lib64/tls/libpthread-2.3.4.so) ==14885== by 0x3963DC68C2: clone (in /lib64/tls/libc-2.3.4.so) ==14885== ==14885== LEAK SUMMARY: ==14885== definitely lost: 235,296 bytes in 1,629 blocks. ==14885== indirectly lost: 7,799,285 bytes in 186,506 blocks. ==14885== possibly lost: 16,468 bytes in 63 blocks. ==14885== still reachable: 105,136 bytes in 2,703 blocks. ==14885== suppressed: 0 bytes in 0 blocks. ==14885== Reachable blocks (those to which a pointer was found) are not shown.
[19 Jan 2009 15:58]
MySQL Verification Team
I couldn't repeat on current source server built on Suse 11.1 64-bit: miguel@hegel:~/dbs> 5.0v/bin/mysql --defaults-file=/home/miguel/dbs/5.0v/my.cnf -uroot Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 Server version: 5.0.77-valgrind-max-debug Source distribution Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> show variables like "%ssl%"; +---------------+----------------------------------+ | Variable_name | Value | +---------------+----------------------------------+ | have_openssl | YES | | have_ssl | YES | | ssl_ca | /home/miguel/dbs/cacert.pem | | ssl_capath | | | ssl_cert | /home/miguel/dbs/server-cert.pem | | ssl_cipher | | | ssl_key | /home/miguel/dbs/server-key.pem | +---------------+----------------------------------+ 7 rows in set (0.32 sec) mysql> exit Bye ==23373== LEAK SUMMARY: ==23373== definitely lost: 0 bytes in 0 blocks. ==23373== possibly lost: 2,448 bytes in 9 blocks. ==23373== still reachable: 1,560 bytes in 5 blocks. ==23373== suppressed: 0 bytes in 0 blocks. --23373-- memcheck: sanity checks: 365 cheap, 14 expensive --23373-- memcheck: auxmaps: 0 auxmap entries (0k, 0M) in use --23373-- memcheck: auxmaps_L1: 0 searches, 0 cmps, ratio 0:10 --23373-- memcheck: auxmaps_L2: 0 searches, 0 nodes --23373-- memcheck: SMs: n_issued = 512 (8192k, 8M) --23373-- memcheck: SMs: n_deissued = 302 (4832k, 4M) --23373-- memcheck: SMs: max_noaccess = 524287 (8388592k, 8191M) --23373-- memcheck: SMs: max_undefined = 306 (4896k, 4M)
[26 Jan 2009 9:14]
Sveta Smirnova
Thank you for the feedback. Please update to current version 5.0.75 (or at least 5.0.67 binaries we provide for), try with it and if problem still exists please send us command line you use to connect to the server, my.cnf, output of command \s in mysql command line client and output of SHOW VARIABLES LIKE '%ssl%'
[26 Jan 2009 14:19]
Liam Kelleher
I have downloaded the source for 5.0.75 and in the code in the acl_getroot() function under the cases for case SSL_TYPE_X509: and case SSL_TYPE_SPECIFIED: in the #ifdef HAVE_OPENSSL section there are calls to SSL_get_peer_certificate() but I do not see where the matching X509_free() is being called after these code paths have being traversed as required by the man page for SSL_get_peer_certificate() i.e. http://openssl.org/docs/ssl/SSL_get_peer_certificate.html I will build and test but based on this and my previous tests I do not expect that the issue is resolved in this version. The issue in the mysqld itself.
[28 Jan 2009 17:48]
Liam Kelleher
I tested with 5.0.75 configured and built with ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --target=x86_64-redhat-linux-gnu --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --with-readline --with-vio --with-openssl --without-debug --enable-shared --with-bench --localstatedir=/var/lib/mysql --with-unix-socket-path=/var/lib/mysql/mysql.sock --with-mysqld-user=mysql --with-extra-charsets=all --with-innodb --with-isam --with-berkeley-db --enable-local-infile --enable-large-files=yes --enable-largefile=yes --enable-thread-safe-client --disable-dependency-tracking --with-named-thread-libs=-lpthread Following this I again see an increasing memory footprint from mysqld. With it starting at ~180MB and constantly increasing moveing to ~500MB after a number of hours with lots of connections being made to the server. Again test changing the code to insert matching X509_free() for each call to SSL_get_peer_certificate() in sql_acl.cc as described previously allowed mysqld to run for a number of hours with the same "load" but with the memory footprint remaining static at ~180MB.
[16 May 2009 22:38]
Andrii Nikitin
Verified from source code and documentation. There are two calls of SSL_get_peer_certificate() in sql\sql_acl.cc which do not have matching X509_free(). In Yassl implementation X509_free() has empty body, so leak exists only in Openssl.
[17 May 2009 0:04]
Andrii Nikitin
Below is my vision of fix for 5.0.79 (same as for other versions)
sql/sql_acl.cc
827,830c827,836
< if (vio_type(vio) == VIO_TYPE_SSL &&
< SSL_get_verify_result(ssl) == X509_V_OK &&
< SSL_get_peer_certificate(ssl))
< user_access= acl_user->access;
---
> if (vio_type(vio) == VIO_TYPE_SSL &&
> SSL_get_verify_result(ssl) == X509_V_OK)
> {
> X509 *cert = SSL_get_peer_certificate(ssl);
> if (cert)
> {
> X509_free(cert);
> user_access= acl_user->access;
> }
> }
879a886
> X509_free(cert);
898a906
> X509_free(cert);
903a912
> X509_free(cert);
[25 May 2009 13:01]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/74888 2909 Davi Arnaut 2009-05-25 Bug#42158: leak: SSL_get_peer_certificate() doesn't have matching X509_free() The problem is that the server failed to follow the rule that every X509 object retrieved using SSL_get_peer_certificate() must be explicitly freed by X509_free(). This caused a memory leak for builds linked against OpenSSL where the X509 object is reference counted -- improper counting will prevent the object from being destroyed once the session containing the peer certificate is freed. The solution is to explicitly free every X509 object used. @ mysql-test/r/openssl_1.result Add test case result for Bug#42158 @ mysql-test/t/openssl_1.test Add test case for Bug#42158 @ sql/sql_acl.cc Deallocate X509 objects.
[25 May 2009 15:11]
Davi Arnaut
Queued to 5.1-bugteam
[28 May 2009 8:19]
Bugs System
Pushed into 5.1.36 (revid:joro@sun.com-20090528073639-yohsb4q1jzg7ycws) (version source revid:davi.arnaut@sun.com-20090525130018-hdqixrfxsk13lvz4) (merge vers: 5.1.36) (pib:6)
[1 Jun 2009 19:33]
Paul DuBois
Noted in 5.1.36 changelog. Builds linked against OpenSSL had a memory leak in association with use of X509 certificates. Setting report to NDI pending push into 6.0.x.
[17 Jun 2009 19:26]
Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090616183122-chjzbaa30qopdra9) (version source revid:davi.arnaut@sun.com-20090525151348-3zcun6oq0cxsbon1) (merge vers: 6.0.12-alpha) (pib:11)
[25 Jun 2009 22:37]
Paul DuBois
Noted in 5.4.4 changelog.
[12 Aug 2009 22:43]
Paul DuBois
Noted in 5.4.2 changelog because next 5.4 version will be 5.4.2 and not 5.4.4.
[15 Aug 2009 2:01]
Paul DuBois
Ignore previous comment about 5.4.2.
[26 Aug 2009 13:46]
Bugs System
Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)
[26 Aug 2009 13:46]
Bugs System
Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)
[26 Aug 2009 13:48]
Bugs System
Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)
[27 Aug 2009 16:33]
Bugs System
Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)
[8 Oct 2009 2:51]
Paul DuBois
The 5.4 fix has been pushed to 5.4.2.
