MySQL Bugs: #62856: Check for "stack overrun" doesn't work with gcc-4.6, server crashes

Bug #62856	Check for "stack overrun" doesn't work with gcc-4.6, server crashes
Submitted:	21 Oct 2011 7:51	Modified:	26 Mar 2013 14:01
Reporter:	Jonas Oreland	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Compiling	Severity:	S3 (Non-critical)
Version:	< 5.5	OS:	Any
Assigned to:	Tor Didriksen	CPU Architecture:	Any

Description:
Similar to bug#42213, where HP compiler was "too smart" or the check of
stack direction.

GCC 4.6 is also too smart, and the test gives incorrect stack direction.
(if compiling with e.g -O3)

In 5.5, the test has been rewritten...

How to repeat:
Compile 5.1 with GCC-4.6 -O3 
(gcc-4.6 is default on fedora15)

Suggested fix:
Backport stack_direction.c from 5.5 into configure.in of earlier releases.

Suggested patch

Attachment: bug62856.patch (text/x-patch), 1.09 KiB.

This patch resolved the testsuite failures we have been experiencing on some Ubuntu Oneiric and Precise configurations:
    main.not_embedded_server
    main.sp_notembedded
    main.subselect_notembedded
    main.execution_constants
The configure log showed that the stack direction was determined wrong there. The patch fixed that. It did not however help with SSL failures (bug 61509). There we see the failures even with the correct stack direction.

In general I believe that guessing stack direction by a C program test is never guaranteed to work.  Even ignoring that the compilers optimize the tests away, the test boils down to some &a < &b test, where a and b are two local vars on different stack frames and that is undefined behaviour in C: pointer greatness test is only defined for pointers pointing to inside of the same object.

Noted in 5.1.70 changelog.

The test for stack overrun did not work for recent gcc versions and
could lead to server exit.

5.1$ bzr log -r 3976.1.1
------------------------------------------------------------
revno: 3976.1.1
committer: Tor Didriksen <tor.didriksen@oracle.com>
branch nick: 5.1
timestamp: Tue 2013-03-26 08:22:45 +0100
message:
  Bug#62856 Check for "stack overrun" doesn't work with gcc-4.6, server crashes 
  Bug#13243248 CHECK FOR "STACK OVERRUN" DOESN'T WORK WITH GCC-4.6, SERVER CRASHES
  
  The existing check for stack direction may give wrong results
  for new versions of gcc at high optimization levels.
  
  Solution: Backport the stack-direction check from 5.5