Bug #61509 mysqld (5.1.57) segfaults with gcc 4.6
Submitted: 14 Jun 2011 9:40 Modified: 29 Jan 2014 0:44
Reporter: Ondej Sur Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Compiling Severity:S1 (Critical)
Version:5.1.57, 5.5.17, 5.5.28 OS:Linux (Debian unstable)
Assigned to: CPU Architecture:Any
Tags: segfault

[14 Jun 2011 9:40] Ondej Sur
Description:
The mysqld tests fails with several segfaults and the build fails.

More detailed report can be found at:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=614044

and

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=627208

(this include the stack traces, etc.)

I have nailed it down to a code which comes from YaSSL and TaoCrypt (example backtrace):

rpl.rpl_ssl/mysqld.2/data/core

(gdb) bt
#0  x86_fallback_frame_state (context=0x5648bb2c, fs=0x5648ba6c) at ../../../src/libgcc/../gcc/config/i386/linux-unwind.h:125
#1  uw_frame_state_for (context=0x5648bb2c, fs=0x5648ba6c) at ../../../src/libgcc/../gcc/unwind-dw2.c:1145
#2  uw_frame_state_for (context=0x5648bb2c, fs=0x5648ba6c) at ../../../src/libgcc/../gcc/unwind-dw2.c:1124
#3  0x55f83dd1 in _Unwind_Backtrace (trace=0x5606b480 <backtrace_helper>, trace_argument=0x5648bbc8) at ../../../src/libgcc/../gcc/unwind.inc:290
#4  0x5606b675 in __backtrace (array=0x5648bc2c, size=128) at ../sysdeps/i386/backtrace.c:127
#5  0x55a8c799 in my_print_stacktrace ()
#6  0x557c33dd in handle_segfault ()
#7  <signal handler called>
#8  0x55ad4c2f in TaoCrypt::ARC4::AsmProcess(unsigned char*, unsigned char const*, unsigned int) ()
#9  0x38a95d3a in ?? ()
#10 0x55ad4d58 in TaoCrypt::ARC4::Process(unsigned char*, unsigned char const*, unsigned int) ()
#11 0x55aebb01 in TaoCrypt::RandomNumberGenerator::GenerateBlock(unsigned char*, unsigned int) ()
#12 0x55aebd0e in TaoCrypt::RandomNumberGenerator::RandomNumberGenerator() ()
#13 0x55acc301 in yaSSL::RandomPool::RandomPool() ()
#14 0x55ac506c in yaSSL::Crypto::Crypto() ()
#15 0x55ac7b76 in yaSSL::SSL::SSL(yaSSL::SSL_CTX*) ()
#16 0x55ab53b9 in yaSSL_new ()
#17 0x55a731e7 in ssl_do ()
#18 0x5593ada9 in mysql_real_connect ()
#19 0x558ecae8 in connect_to_master(THD*, st_mysql*, Master_info*, bool, bool) ()
#20 0x558f32c4 in handle_slave_io ()
#21 0x55dd9c39 in start_thread (arg=0x5648db70) at pthread_create.c:304
#22 0x560568ee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

How to repeat:
Compile with gcc-4.6.

Suggested fix:
Compilation with gcc-4.5 works just fine.
[9 Nov 2011 20:54] Clint Byrum
This still affects 5.5.17 btw, exact same result.
[6 Dec 2011 10:00] Bjoern Boschman
is there a way to assign this ticket also to 5.5 branch and if applicable also to 5.6 branch to raise the Severity also into more current versions.

I think that this could speed up things
[6 Dec 2011 13:15] Jonas Oreland
how about patch attached to http://bugs.mysql.com/bug.php?id=62856 ?
[19 Jan 2012 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[19 Jan 2012 3:10] Clint Byrum
As I stated, this was seen in 5.5.17, not just 5.1, so I don't think that this is a duplicate of http://bugs.mysql.com/bug.php?id=62856
[20 Jan 2012 13:56] Valeriy Kravchuk
Clone off date for 5.5.17 was October 19, 2011, 2 days before that other bug was reported. So, the fact that 5.5.17 is also affected proves nothing. Had you tried the patch, or recent 5.5.20 version (it should NOT be affected)?
[21 Feb 2012 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[25 Aug 2012 8:24] Laurynas Biveinis
The problem here does not seem to be the stack direction issue (although it might pile up additional issues on the top), but rather the IMHO broken inline assemly in taocrypt arc4.cpp: (only interesting parts)

__attribute__ ((noinline))
void ARC4::AsmProcess(byte* out, const byte* in, word32 length)
{
    AS2(    mov    [esp], ebp               )   // length
}

This assumes a certain calling convention and does not expose it to the compiler (attribute((noinline)) might be a poor attempt at that). The compiler cannot even tell if the args are used there. The proper way to write this function is to use the GCC extended asm syntax to pass the args the to the inline asm. I'm testing a workaround of adding -fno-omit-frame-pointer to taocrypt/src/Makefile.am, this might buy some time before the next compiler version breaks things again.
[27 Aug 2012 10:23] Laurynas Biveinis
We see this on 5.1 only. Before, 30 SSL test crashes:

http://jenkins.percona.com/job/percona-server-5.1-trunk/619/BUILD_TYPE=debug,Host=ubuntu-o...
http://jenkins.percona.com/job/percona-server-5.1-trunk/619/BUILD_TYPE=debug,Host=ubuntu-o...

After, no SSL test crashes:

http://jenkins.percona.com/job/percona-server-5.1-param/395/BUILD_TYPE=debug,Host=ubuntu-o...
http://jenkins.percona.com/job/percona-server-5.1-param/395/BUILD_TYPE=debug,Host=ubuntu-o...

The workaround:

=== modified file 'Percona-Server/extra/yassl/taocrypt/src/Makefile.am'
--- Percona-Server/extra/yassl/taocrypt/src/Makefile.am	2012-02-13 11:40:12 +0000
+++ Percona-Server/extra/yassl/taocrypt/src/Makefile.am	2012-08-27 05:43:12 +0000
@@ -24,7 +24,7 @@
         tftables.cpp twofish.cpp crypto.cpp rabbit.cpp hc128.cpp
 
 libtaocrypt_la_CXXFLAGS = @yassl_taocrypt_extra_cxxflags@ -DYASSL_PURE_C \
-			  @yassl_thread_cxxflags@
+			  @yassl_thread_cxxflags@ -fno-omit-frame-pointer
 
 EXTRA_DIST = $(wildcard ../include/*.hpp)
[29 Jan 2014 0:44] Paul DuBois
Noted in 5.5.7 changelog.

mysqld segfaulted if compiled with gcc 4.6.
[27 Mar 2014 14:22] Laurynas Biveinis
Per my comment on 25 Aug 2012 8:24, the inline asm in taocrypt was broken as the input and output variables were not marked as such, thus the compiler was free to miscompile.

This has been fixed with (didn't check whether fully or partially) with

5.5$ bzr log -r 4594 -n0
------------------------------------------------------------
revno: 4594
committer: Anirudh Mangipudi <anirudh.mangipudi@oracle.com>
branch nick: mysql-5.5
timestamp: Thu 2014-02-06 11:16:55 +0530
message:
  Bug#14211271 ISSUES WITH SSL ON DEBIAN WHEEZY I386 AND KFREEBSD-I386
  
  Problem:
  It was reported that on Debian and KFreeBSD platforms, i386 architecture 
  machines certain SSL tests are failing. main.ssl_connect  rpl.rpl_heartbeat_ssl
  rpl.rpl_ssl1 rpl.rpl_ssl main.ssl_cipher, main.func_encrypt were the tests that
   were reportedly failing (crashing). The reason for the crashes are said to be
  due to the assembly code of yaSSL.
  
  Solution:
  There was initially a workaround suggested i.e., to enable 
  -DTAOCRYPT_DISABLE_X86ASM flag which would prevent the crash, but at an expense
   of 4X reduction of speed. Since this was unacceptable, the fix was the 
  functions using assembly, now input variables from the function call using 
  extended inline assembly on GCC instead of relying on direct assembly code.