Bug #62856 Check for "stack overrun" doesn't work with gcc-4.6, server crashes
Submitted: 21 Oct 2011 7:51 Modified: 26 Mar 2013 14:01
Reporter: Jonas Oreland Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Compiling Severity:S3 (Non-critical)
Version:< 5.5 OS:Any
Assigned to: Tor Didriksen CPU Architecture:Any
Triage: Needs Triage: D3 (Medium)

[21 Oct 2011 7:51] Jonas Oreland
Description:
Similar to bug#42213, where HP compiler was "too smart" or the check of
stack direction.

GCC 4.6 is also too smart, and the test gives incorrect stack direction.
(if compiling with e.g -O3)

In 5.5, the test has been rewritten...

How to repeat:
Compile 5.1 with GCC-4.6 -O3 
(gcc-4.6 is default on fedora15)

Suggested fix:
Backport stack_direction.c from 5.5 into configure.in of earlier releases.
[21 Oct 2011 9:29] Jonas Oreland
Suggested patch

Attachment: bug62856.patch (text/x-patch), 1.09 KiB.

[25 Aug 2012 7:15] Laurynas Biveinis
This patch resolved the testsuite failures we have been experiencing on some Ubuntu Oneiric and Precise configurations:
    main.not_embedded_server
    main.sp_notembedded
    main.subselect_notembedded
    main.execution_constants
The configure log showed that the stack direction was determined wrong there. The patch fixed that. It did not however help with SSL failures (bug 61509). There we see the failures even with the correct stack direction.

In general I believe that guessing stack direction by a C program test is never guaranteed to work.  Even ignoring that the compilers optimize the tests away, the test boils down to some &a < &b test, where a and b are two local vars on different stack frames and that is undefined behaviour in C: pointer greatness test is only defined for pointers pointing to inside of the same object.
[26 Mar 2013 14:01] Paul Dubois
Noted in 5.1.70 changelog.

The test for stack overrun did not work for recent gcc versions and
could lead to server exit.
[26 Jun 2013 9:01] Laurynas Biveinis
5.1$ bzr log -r 3976.1.1
------------------------------------------------------------
revno: 3976.1.1
committer: Tor Didriksen <tor.didriksen@oracle.com>
branch nick: 5.1
timestamp: Tue 2013-03-26 08:22:45 +0100
message:
  Bug#62856 Check for "stack overrun" doesn't work with gcc-4.6, server crashes 
  Bug#13243248 CHECK FOR "STACK OVERRUN" DOESN'T WORK WITH GCC-4.6, SERVER CRASHES
  
  The existing check for stack direction may give wrong results
  for new versions of gcc at high optimization levels.
  
  Solution: Backport the stack-direction check from 5.5