Bug #21250 resolve stack traces on x86-64 (AMD64, EM64T): most 64 bit CPUs we see
Submitted: 24 Jul 2006 13:19 Modified: 26 Sep 2008 15:49
Reporter: Elliot Murphy Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Errors Severity:S3 (Non-critical)
Version:5.0.68 OS:Linux (64-bit)
Assigned to: Paul Dubois CPU Architecture:Any
Tags: bfsm_2007_08_16
Triage: D1 (Critical) / R4 (High) / E2 (Low)

[24 Jul 2006 13:19] Elliot Murphy
Description:
http://www.mysqlperformanceblog.com/2006/07/19/stack-trace-for-x86_64-boxes/

This is a patch from Vadim to automatically resolve stack during a crash on x86_64 (currently MySQL only does this on x86).

How to repeat:
look at a crash on amd64

Suggested fix:
apply the patch.
[25 Aug 2006 14:00] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/10885

ChangeSet@1.2252, 2006-08-25 17:59:47+04:00, kaa@polly.local +2 -0
  Added stacktrace dumps for x86_64 (bug #21250)
  Fixed stacktrace dumps for i386/NPTL
[26 Aug 2006 9:42] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/10903

ChangeSet@1.2253, 2006-08-26 10:21:26+04:00, kaa@polly.local +1 -0
  Post-review fix (bug #21250)
[31 Aug 2006 11:45] Magnus BlÄudd
Pushed to 5.0.25
[2 Sep 2006 2:08] Paul Dubois
Noted in 5.0.25 changelog.
[6 Sep 2006 21:59] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/11501

ChangeSet@1.2538, 2006-09-07 00:01:00+02:00, tsmith@maint1.mysql.com +2 -0
  Bug #21250: esolve stack traces on AMD64 (backport to mysql-4.1)
[7 Sep 2006 17:07] Timothy Smith
Note: this patch has been back-ported into 4.1.

Queued in the -maint team tree, will be merged into global 4.1 and available in next 4.1 release (4.1.22).
[7 Sep 2006 18:21] Paul Dubois
Noted in 4.1.22 changelog.
[4 Oct 2006 13:56] Chad MILLER
Available in 4.1.22.
[11 May 2007 9:10] Shane Bester
this patch doesn't seem to work in 99% of the cases.
we're not getting any stack traces on 64-bit binaries.
in about 50% of cases the query is printed, but stack trace always contains only a single line with what appears to be ascii characters.

setting back to verified status.
[29 Jul 2008 5:51] Shane Bester
please also fix the thd->query output. 99% of the time this not even a valid pointer.  there are some cases where it makes sense, such as replication thread/rbr, or background threads, but we've seen this value corrupted alot even on normal queries.
[7 Aug 2008 12:12] Shane Bester
related to this bug is bug #35987 which applies to windows binaries.
[10 Sep 2008 6:49] Shane Bester
Noted, 5.1.28 seems to work but 5.0.68 doesn't work in this case:

login to ndbsup-1 and run the following two testcases
against the mysql-enterprise-gpl-5.1.28-rc-linux-x86_64-glibc23

create table t (t timestamp) engine=innodb;
select * from t where t>'2008-01-01' and t='0000-00-00';

prepare s from 'select 1 from dual where (@@new)';
execute s;

Here's the error log!

080910  8:40:12 [Note] /users/sbester/bug21250/mysql-enterprise-gpl-5.1.28-rc-linux-x86_64-glibc23/mysql-enterprise-gpl-5.0.68-linux-x86_64-glibc23/bin/mysqld: ready for connections.
Version: '5.0.68-enterprise-gpl'  socket: 'sock'  port: 3333  MySQL Enterprise Server (GPL)
080910  8:40:24 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225787 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x11586b0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x44688118, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
New value of fp=0x11586b0 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x116e050 = select * from t where t>'2008-01-01' and
t='0000-00-00'
thd->thread_id=1
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
080910 08:40:24  mysqld restarted
InnoDB: Log scan progressed past the checkpoint lsn 0 43655
080910  8:40:25  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 0 45141
080910  8:40:25  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
080910  8:40:25  InnoDB: Started; log sequence number 0 45141
080910  8:40:25 [Note] /users/sbester/bug21250/mysql-enterprise-gpl-5.1.28-rc-linux-x86_64-glibc23/mysql-enterprise-gpl-5.0.68-linux-x86_64-glibc23/bin/mysqld: ready for connections.
Version: '5.0.68-enterprise-gpl'  socket: 'sock'  port: 3333  MySQL Enterprise Server (GPL)
mysqld: my_new.cc:50: int __cxa_pure_virtual(): Assertion `"Pure virtual method called." == "Aborted"' failed.
080910  8:40:33 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225787 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x1154b90
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x45089118, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
(nil)
New value of fp=0x1154b90 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x116fec8 = select 1 from dual where (@@new)
thd->thread_id=1
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
080910 08:40:33  mysqld restarted
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
080910  8:40:33  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
080910  8:40:34  InnoDB: Started; log sequence number 0 45141
080910  8:40:34 [Note] /users/sbester/bug21250/mysql-enterprise-gpl-5.1.28-rc-linux-x86_64-glibc23/mysql-enterprise-gpl-5.0.68-linux-x86_64-glibc23/bin/mysqld: ready for connections.
Version: '5.0.68-enterprise-gpl'  socket: 'sock'  port: 3333  MySQL Enterprise Server (GPL)
[10 Sep 2008 6:51] Shane Bester
removing 5.1 from the versions affected until I see more recent 5.1 error logs without stack trace on systems that support it.
[26 Sep 2008 15:49] Paul Dubois
Amended changelog entry:

MySQL did not properly do stack dumps on x86_64 and i386/NPTL
systems. (Note that the initial fix for this problem was discovered
not to be correct. Further work on the problem was undertaken only
for MySQL 5.1 and up. See Bug#31891.)