| Bug #34236 | Various possibly related SSL crashes | ||
|---|---|---|---|
| Submitted: | 1 Feb 2008 15:40 | Modified: | 14 Oct 2010 13:16 |
| Reporter: | Michael Näf | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Compiling | Severity: | S1 (Critical) |
| Version: | 5.0.51a, 5.1.45, 5.1.47 | OS: | Any |
| Assigned to: | Davi Arnaut | CPU Architecture: | Any |
| Tags: | SSL | ||
[1 Feb 2008 15:40]
Michael Näf
[1 Feb 2008 15:42]
Michael Näf
changed severity to critical because the bug causes the server to crash. Note that the server restarts automatically and then runs reliably again for millions of queries until the next crash.
[1 Feb 2008 17:57]
Valeriy Kravchuk
Thank you for a problem report. Please, try to repeat with a newer version, 5.0.51a, and inform about the results.
[2 Mar 2008 0:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[17 Mar 2008 10:26]
Susanne Ebrecht
We still need to know if you have this issues by using our newest version 5.0.51a.
[17 Mar 2008 10:57]
Michael Näf
I cannot say because I cannot update because we run Debian and our policy is to only upgrade packages when they are officially published by Debian. Debian's latest MySQL version is 5.0.32.
[19 Mar 2008 17:58]
Susanne Ebrecht
That's not true. Debian already made packages for MySQL 5.0.51a. Please use backports for installing them.
[25 Mar 2008 0:24]
Michael Näf
Thanks for the hint. I upgraded a minute ago.
[25 Mar 2008 22:13]
Michael Näf
Problem also occurs with version 5.0.51a (changed version attribute of this case). Syslog: Mar 25 22:00:41 D08L0801 mysqld[2487]: 080325 22:00:41 - mysqld got signal 11; Mar 25 22:00:41 D08L0801 mysqld[2487]: This could be because you hit a bug. It is also possible that this binary Mar 25 22:00:41 D08L0801 mysqld[2487]: or one of the libraries it was linked against is corrupt, improperly built, Mar 25 22:00:41 D08L0801 mysqld[2487]: or misconfigured. This error can also be caused by malfunctioning hardware. Mar 25 22:00:41 D08L0801 mysqld[2487]: We will try our best to scrape up some info that will hopefully help diagnose Mar 25 22:00:41 D08L0801 mysqld[2487]: the problem, but since we have already crashed, something is definitely wrong Mar 25 22:00:41 D08L0801 mysqld[2487]: and this may fail. Mar 25 22:00:41 D08L0801 mysqld[2487]: Mar 25 22:00:41 D08L0801 mysqld[2487]: key_buffer_size=2147483648 Mar 25 22:00:41 D08L0801 mysqld[2487]: read_buffer_size=131072 Mar 25 22:00:41 D08L0801 mysqld[2487]: max_used_connections=203 Mar 25 22:00:41 D08L0801 mysqld[2487]: max_connections=400 Mar 25 22:00:41 D08L0801 mysqld[2487]: threads_connected=102 Mar 25 22:00:41 D08L0801 mysqld[2487]: It is possible that mysqld could use up to Mar 25 22:00:41 D08L0801 mysqld[2487]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2967548 K Mar 25 22:00:41 D08L0801 mysqld[2487]: bytes of memory Mar 25 22:00:41 D08L0801 mysqld[2487]: Hope that's ok; if not, decrease some variables in the equation. Mar 25 22:00:41 D08L0801 mysqld[2487]: Mar 25 22:00:41 D08L0801 mysqld[2487]: thd=0x26ada6f0 Mar 25 22:00:41 D08L0801 mysqld[2487]: Attempting backtrace. You can use the following information to find out Mar 25 22:00:41 D08L0801 mysqld[2487]: where mysqld died. If you see no messages after this, something went Mar 25 22:00:41 D08L0801 mysqld[2487]: terribly wrong... Mar 25 22:00:41 D08L0801 mysqld[2487]: Cannot determine thread, fp=0x29addd48, backtrace may not be correct. Mar 25 22:00:41 D08L0801 mysqld[2487]: Stack range sanity check OK, backtrace follows: Mar 25 22:00:41 D08L0801 mysqld[2487]: 0x81cd76d Mar 25 22:00:41 D08L0801 mysqld[2487]: 0x85047b7 Mar 25 22:00:41 D08L0801 mysqld[2487]: 0x84fee4a Mar 25 22:00:41 D08L0801 mysqld[2487]: 0x8500f77 Mar 25 22:00:41 D08L0801 mysqld[2487]: 0x85131a0 Mar 25 22:00:41 D08L0801 mysqld[2487]: 0x85133a8 Mar 25 22:00:41 D08L0801 mysqld[2487]: 0x84f5e3b Mar 25 22:00:41 D08L0801 mysqld[2487]: 0x84ba266 Mar 25 22:00:41 D08L0801 mysqld[2487]: 0x81eb529 Mar 25 22:00:41 D08L0801 mysqld[2487]: 0xb7eb9240 Mar 25 22:00:41 D08L0801 mysqld[2487]: 0xb7cf449e Mar 25 22:00:41 D08L0801 mysqld[2487]: New value of fp=(nil) failed sanity check, terminating stack trace! Mar 25 22:00:41 D08L0801 mysqld[2487]: Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved Mar 25 22:00:41 D08L0801 mysqld[2487]: stack trace is much more helpful in diagnosing the problem, so please do Mar 25 22:00:41 D08L0801 mysqld[2487]: resolve it Mar 25 22:00:41 D08L0801 mysqld[2487]: Trying to get some variables. Mar 25 22:00:41 D08L0801 mysqld[2487]: Some pointers may be invalid and cause the dump to abort... Mar 25 22:00:41 D08L0801 mysqld[2487]: thd->query at (nil) is invalid pointer Mar 25 22:00:41 D08L0801 mysqld[2487]: thd->thread_id=26505 Mar 25 22:00:41 D08L0801 mysqld[2487]: The manual page at http://www.mysql.com/doc/en/Crashing.html contains Mar 25 22:00:41 D08L0801 mysqld[2487]: information that should help you find out what is causing the crash. Mar 25 22:00:41 D08L0801 mysqld_safe[1605]: Number of processes running now: 0 Mar 25 22:00:41 D08L0801 mysqld_safe[1607]: restarted Mar 25 22:00:42 D08L0801 mysqld[1610]: 080325 22:00:42 InnoDB: Started; log sequence number 0 43685 Mar 25 22:00:42 D08L0801 mysqld[1610]: 080325 22:00:42 [Note] Recovering after a crash using /var/log/mysql/mysql-bin Mar 25 22:00:42 D08L0801 mysqld[1610]: 080325 22:00:42 [Note] Starting crash recovery... Mar 25 22:00:42 D08L0801 mysqld[1610]: 080325 22:00:42 [Note] Crash recovery finished. Mar 25 22:00:43 D08L0801 mysqld[1610]: 080325 22:00:43 [Note] /usr/sbin/mysqld: ready for connections. Mar 25 22:00:43 D08L0801 mysqld[1610]: Version: '5.0.51a-3-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Debian) Backtrace: 0x81cd76d handle_segfault + 797 0x85047b7 _ZNK5yaSSL11SSL_SESSION9GetBornOnEv + 7 0x84fee4a _ZN5yaSSL11ClientHello7ProcessERNS_12input_bufferERNS_3SSLE + 490 0x8500f77 _ZN5yaSSL15HandShakeHeader7ProcessERNS_12input_bufferERNS_3SSLE + 279 0x85131a0 _ZN5yaSSL14DoProcessReplyERNS_3SSLE + 576 0x85133a8 _ZN5yaSSL12processReplyERNS_3SSLE + 56 0x84f5e3b .L426 + 8 0x84ba266 ssl_do + 118 0x81eb529 handle_one_connection + 873 0xb7eb9240 _end + -1351080624 0xb7cf449e _end + -1352935506 if you don't have c++filt, the next command will fail /usr/bin/resolve_stack_dump -s ./symbols -n ./stack | c++filt 0x81cd76d handle_segfault + 797 0x85047b7 yaSSL::SSL_SESSION::GetBornOn() const + 7 0x84fee4a yaSSL::ClientHello::Process(yaSSL::input_buffer&, yaSSL::SSL&) + 490 0x8500f77 yaSSL::HandShakeHeader::Process(yaSSL::input_buffer&, yaSSL::SSL&) + 279 0x85131a0 yaSSL::DoProcessReply(yaSSL::SSL&) + 576 0x85133a8 yaSSL::processReply(yaSSL::SSL&) + 56 0x84f5e3b .L426 + 8 0x84ba266 ssl_do + 118 0x81eb529 handle_one_connection + 873 0xb7eb9240 _end + -1351080624 0xb7cf449e _end + -1352935506
[14 Apr 2008 11:03]
Michael Näf
Can you give me a hint as to whether this is likely to be an SSL issue, or a hardware issue, or something else completely?
[24 Apr 2008 15:42]
Michael Näf
The problem goes away when I stop using SSL to connect to the server.
[10 Mar 2009 13:32]
Ben Wallach
Looks like it happened today again. It occurs when one of the scripts calls OPTIMIZE TABLE....then server goes down.
[24 Aug 2009 10:15]
Sveta Smirnova
Ben, thank you for the feedback. Backtrace which you provided looks different from original one. Is it related to SSL connection? If yes, please upload table which caused crash to our FTP server and add comment here with filename. Otherwise please open new bug report and upload table which caused crash as well.
[8 Nov 2009 19:01]
Valeriy Kravchuk
All reproters: What version of yaSSL do you have?
[9 Dec 2009 0:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[11 May 2010 9:31]
Dmitry Kolesnik
I have the similar problem. I have posted a new bug here http://bugs.mysql.com/bug.php?id=53287
[18 May 2010 2:54]
Roel Van de Paar
There are several different SSL crashes which may be related: #1 _ZN5mySTL7find_ifINS_4listIPN5yaSSL11SSL_SESSIONEE8iteratorENS2_20yassl_int_cpp_local210sess_matchEEET_S9_S9_T0_ Listed in this bug, in bug #53287 and in bug #48239 #2 _ZNK5yaSSL11SSL_SESSION9GetBornOnEv / mysqld.exe!yaSSL::SSL_SESSION::GetBornOn()[yassl_int.cpp:1522] Listed in this bug, in bug #53287 and in bug #48239 #3 ntdll.dll!RtlFreeHeap() > kernel32.dll!HeapFree() > mysqld.exe!free()[free.c:110] > mysqld.exe!mySTL::list<yaSSL::output_buffer * __ptr64>::pop_front()[list.hpp:246] > mysqld.exe!yaSSL::SSL::flushBuffer()[yassl_int.cpp:1123] Listed in bug #53287 alone. Adding it here to make this bug complete: ============ 100428 1:00:02 - mysqld got exception 0xc0000005 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=209715200 read_buffer_size=2097152 max_used_connections=23 max_threads=10000 threads_connected=8 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1958544 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd: 0x3a51c7d0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... 0000000077648D50 ntdll.dll!RtlFreeHeap() 00000000774FC4CA kernel32.dll!HeapFree() 000000014000581B mysqld.exe!free()[free.c:110] 000000014035A00F mysqld.exe!mySTL::list<yaSSL::output_buffer * __ptr64>::pop_front()[list.hpp:246] 000000014035A8BA mysqld.exe!yaSSL::SSL::flushBuffer()[yassl_int.cpp:1123] 000000014034E2B9 mysqld.exe!yaSSL_accept()[ssl.cpp:356] 0000000140383B4A mysqld.exe!ssl_do()[viossl.c:205] 0000000140097D7F mysqld.exe!check_connection()[sql_connect.cc:805] 000000014009804E mysqld.exe!login_connection()[sql_connect.cc:956] 00000001400981B7 mysqld.exe!handle_one_connection()[sql_connect.cc:1119] 000000014031AE95 mysqld.exe!pthread_start()[my_winthread.c:85] 00000001402E4F67 mysqld.exe!_callthreadstart()[thread.c:295] 00000001402E5035 mysqld.exe!_threadstart()[thread.c:275] 00000000774EBE3D kernel32.dll!BaseThreadInitThunk() 0000000077626A51 ntdll.dll!RtlUserThreadStart() Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0000000000000000=(null) thd->thread_id=8962 thd->killed=NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. ============ I marked bug #53287 and bug #48239 as duplicates of this one for a better tracking/more easy resolution. Developers: please refer to the individual bugs for additional information if needed.
[26 May 2010 14:46]
MySQL Verification Team
Repeated a crash in testing, attached is relevant debug infos.
Attachment: bug34236_5.1.47_yassl_crash_infos.txt (text/plain), 6.89 KiB.
[26 May 2010 16:15]
MySQL Verification Team
repeated the BornOn crash.. Multiple threads were doing a Flush(), thereby corrupting the session list ?
Attachment: bug34236_5.1.47_yassl_BornOn_crash_infos.txt (text/plain), 7.01 KiB.
[26 May 2010 16:43]
MySQL Verification Team
I don't think this construct is actually doing anything useful. Lock guard(mutex_); So we have lists getting corrupted by multiple concurrent actions. It was obviously designed to call pthread_mutex_lock/or EnterCriticalSection on windows, but examining the assembler proves this is not even being compiled in.
[26 May 2010 17:37]
MySQL Verification Team
bug found! we don't define MULTI_THREADED in our builds. so those locking semantics in yassl are no-ops. if anybody wants to test this by rebuilding with -DMULTI_THREADED and see if it helps, please report results here.
[26 May 2010 18:48]
MySQL Verification Team
This problem is most likely to happen after the Sessions::Flush() call which is triggered every SESSION_FLUSH_COUNT (aka 256) SSL connections. If a new connection comes into the server while the Flush() is being called a problem can occur during or after that.
[27 May 2010 9:30]
Dmitry Kolesnik
Shane Bester , I'm ready to test "if anybody wants to test this by rebuilding with -DMULTI_THREADED ". But we use MySQL under Windows Server2008 (64 bit), Could you please provide the build with fix for testing?
[1 Jun 2010 12:42]
Alex Elkin
Hi! We got exactly the same issue. Could you please provide Windows installer with the fix - or any other way for us to have it resolved. Thanks a lot!
[4 Jun 2010 17:35]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/110270 2867 Davi Arnaut 2010-06-04 Bug#34236: Various possibly related SSL crashes The problem was that the bundled yaSSL library was being built without thread safety support regardless of the thread safeness of the compoments linked with it. The solution is to enable yaSSL thread safety support if any component (server or client) is to be built with thread support. @ config/ac-macros/yassl.m4 Enable yaSSL thread safety if with server or client thread safe. @ extra/yassl/CMakeLists.txt Always enable for Windows builds. @ extra/yassl/include/lock.hpp Rename MULTI_THREAD to YASSL_THREAD_SAFE. @ extra/yassl/src/Makefile.am Use CXXFLAGS to set thread related definitions as the lock header has no local dependencies. @ extra/yassl/src/lock.cpp Rename MULTI_THREAD to YASSL_THREAD_SAFE. @ extra/yassl/taocrypt/CMakeLists.txt Always enable for Windows builds. @ extra/yassl/taocrypt/benchmark/Makefile.am Pass thread related CXXFLAGS. @ extra/yassl/taocrypt/src/Makefile.am Pass thread related CXXFLAGS. @ extra/yassl/taocrypt/test/Makefile.am Pass thread related CXXFLAGS. @ extra/yassl/taocrypt/test/memory.cpp Rename MULTI_THREAD to YASSL_THREAD_SAFE. @ extra/yassl/testsuite/Makefile.am Pass thread related CXXFLAGS.
[4 Jun 2010 17:38]
Alex Elkin
This is great! Do you have any estimate when the new version may be available? This issue is rather urgent for our production system. Thanks again.
[4 Jun 2010 19:18]
Davi Arnaut
It is going to take some time as 5.1.48 has already been cloned for release. If you are a customer, you have the option to request a special build with this patch.
[5 Jun 2010 10:20]
Alex Elkin
Davi, do you have the time estimate for the next release (with this fix)?
[5 Jun 2010 20:07]
Davi Arnaut
Alex, as this fix didn't make it in time for 5.1.48, a release including this fix is probably a few months away. If you need a binary with this fix ASAP, the fastest way is building it yourself or requesting a new binary via MySQL's technical support or a third-party.
[8 Jun 2010 13:37]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/110503 2870 Davi Arnaut 2010-06-08 Bug#34236: Various possibly related SSL crashes The problem was that the bundled yaSSL library was being built without thread safety support regardless of the thread safeness of the compoments linked with it. The solution is to enable yaSSL thread safety support if any component (server or client) is to be built with thread support. Also, generate new certificates for yaSSL's test suite. @ config/ac-macros/yassl.m4 Enable yaSSL thread safety if linking with the server or a thread safe client library. Avoids building a thread safe yaSSL when only building a non-thread safe client library. @ extra/yassl/CMakeLists.txt Always enable for Windows builds. @ extra/yassl/certs/ca-cert.pem New certificate, previous one expired. @ extra/yassl/certs/client-cert.der New certificate, previous one expired. @ extra/yassl/certs/client-cert.pem New certificate, previous one expired. @ extra/yassl/certs/dsa-cert.pem New certificate, previous one expired. @ extra/yassl/certs/server-cert.pem New certificate, previous one expired. @ extra/yassl/include/lock.hpp Rename MULTI_THREAD to YASSL_THREAD_SAFE. @ extra/yassl/src/Makefile.am Use CXXFLAGS to set thread related definitions as the lock header (lock.hpp) has no local dependencies. @ extra/yassl/src/lock.cpp Rename MULTI_THREAD to YASSL_THREAD_SAFE. @ extra/yassl/taocrypt/CMakeLists.txt Always enable for Windows builds. @ extra/yassl/taocrypt/benchmark/Makefile.am Pass thread related CXXFLAGS. @ extra/yassl/taocrypt/src/Makefile.am Pass thread related CXXFLAGS. @ extra/yassl/taocrypt/test/Makefile.am Pass thread related CXXFLAGS. @ extra/yassl/taocrypt/test/memory.cpp Rename MULTI_THREAD to YASSL_THREAD_SAFE. @ extra/yassl/testsuite/Makefile.am Pass thread related CXXFLAGS.
[8 Jun 2010 21:02]
Davi Arnaut
Queued to mysql-5.0-bugteam and up.
[10 Jun 2010 0:31]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/110662 2872 Davi Arnaut 2010-06-09 Bug#34236: Various possibly related SSL crashes Addendum: Work around a compilation failure on Windows due to windows.h not being added to the global namespace. @ extra/yassl/include/lock.hpp Move windows.h inclusion into the global namespace.
[14 Jun 2010 14:14]
Davi Arnaut
A unrelated note. One might attempt to work around this bug by using stunnel to redirect the server port to a local non-ssl port of the server.
[16 Jun 2010 6:48]
Sveta Smirnova
Bug #54541 was marked as duplicate of this one.
[17 Jun 2010 6:14]
Bugs System
Pushed into 5.5.5-m3 (revid:alexey.kopytov@sun.com-20100615145247-8bj0vmuqlotbqsn9) (version source revid:davi.arnaut@sun.com-20100610004140-8wl59up3lfb3t8bl) (merge vers: 5.5.5-m3) (pib:16)
[17 Jun 2010 6:18]
Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100615150216-cubqoyn1fj9b6a2p) (version source revid:alik@ibmvm-20100610103252-to5miyvcox8lnmgx) (pib:16)
[17 Jun 2010 7:38]
MC Brown
A note has been added to the 5.5.5 changelog: <command>mysqld</command> could fail during execution when using SSL.
[8 Jul 2010 11:54]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/113120 3410 sunanda 2010-07-08 Backport the fix of bug#34236 into the sources of 5.1.48, to create custom build BR#378879. Original comments: > revno: 1810.3987.24 > committer: Davi Arnaut <Davi.Arnaut@Sun.COM> > branch nick: 34236-5.0 > timestamp: Tue 2010-06-08 10:36:47 -0300 > message: > Bug#34236: Various possibly related SSL crashes > > The problem was that the bundled yaSSL library was being built > without thread safety support regardless of the thread safeness > of the compoments linked with it. > revno: 1810.3987.26 > committer: Davi Arnaut <Davi.Arnaut@Sun.COM> > branch nick: 34236-5.0 > timestamp: Wed 2010-06-09 21:30:41 -0300 > message: > Bug#34236: Various possibly related SSL crashes > > Addendum: Work around a compilation failure on Windows due to > windows.h not being added to the global namespace. In addition, the version string got the build request added.
[19 Jul 2010 14:37]
Bugs System
Pushed into 5.1.49 (revid:build@mysql.com-20100719143034-omcma40sblwmay3x) (version source revid:davi.arnaut@sun.com-20100610003620-f60mfhg4o4xkvu52) (merge vers: 5.1.48) (pib:16)
[20 Jul 2010 13:21]
MC Brown
Changelog entry has been added to the 5.1.49 changelog.
[2 Aug 2010 7:49]
Bugs System
Pushed into 5.0.92 (revid:georgi.kodinov@oracle.com-20100802074824-5201e4ppst9t3yqt) (version source revid:davi.arnaut@sun.com-20100610003041-j5lq6241he0lp7eh) (merge vers: 5.0.92) (pib:18)
[2 Aug 2010 13:02]
MC Brown
Added to 5.0.92 changelog
[14 Oct 2010 8:32]
Bugs System
Pushed into mysql-5.1-telco-7.0 5.1.51-ndb-7.0.20 (revid:martin.skold@mysql.com-20101014082627-jrmy9xbfbtrebw3c) (version source revid:vasil.dimov@oracle.com-20100531152341-x2d4hma644icamh1) (merge vers: 5.5.5-m3) (pib:21)
[14 Oct 2010 8:47]
Bugs System
Pushed into mysql-5.1-telco-6.3 5.1.51-ndb-6.3.39 (revid:martin.skold@mysql.com-20101014083757-5qo48b86d69zjvzj) (version source revid:vasil.dimov@oracle.com-20100531152341-x2d4hma644icamh1) (merge vers: 5.5.5-m3) (pib:21)
[14 Oct 2010 9:01]
Bugs System
Pushed into mysql-5.1-telco-6.2 5.1.51-ndb-6.2.19 (revid:martin.skold@mysql.com-20101014084420-y54ecj85j5we27oa) (version source revid:vasil.dimov@oracle.com-20100531152341-x2d4hma644icamh1) (merge vers: 5.5.5-m3) (pib:21)
[14 Oct 2010 13:16]
Jon Stephens
Already documented in the 5.1.49 changelog; no additional changelog entries required. Closed.
