Bug #41549 MySQL crashes with segmentation fault in ha_heap::rnd_next()

• Submitted: 17 Dec 2008 14:27
• Modified: 12 Feb 2009 12:00
• Reporter: Robert Heinzmann
• Status: No Feedback
• Category: MySQL Server: Memory storage engine
• Severity: S2 (Serious)
• Version: 4.0.27-max-log
• OS: Linux (Debian Etch)
• CPU Architecture: Any

[17 Dec 2008 14:27] Robert Heinzmann

Description:
The mysql server randomly crashes at different querys always at the same instruction pointer. Hardware (except disk) has been changed. Filesystem has been checked.

(filtered=resolve_stack_dump -s $SYMS -n $i | c++filt)

/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244

------------------------
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
/var/log/mysql.err.filtered-(nil)
/var/log/mysql.err.filtered-0xb7fbb267 _end + -1346516637
/var/log/mysql.err.filtered-0xb7ed260e _end + -1347470070
/var/log/mysql.err.filtered-New value of fp=(nil) failed sanity check, terminating stack trace!
/var/log/mysql.err.filtered-Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
/var/log/mysql.err.filtered-stack trace is much more helpful in diagnosing the problem, so please do
/var/log/mysql.err.filtered-resolve it
/var/log/mysql.err.filtered-Trying to get some variables.
/var/log/mysql.err.filtered-Some pointers may be invalid and cause the dump to abort...
/var/log/mysql.err.filtered-thd->query at (nil)  is invalid pointer
/var/log/mysql.err.filtered-thd->thread_id=1167
/var/log/mysql.err.filtered-The manual page at http://www.mysql.com/doc/en/Crashing.html contains
/var/log/mysql.err.filtered-information that should help you find out what is causing the crash.
/var/log/mysql.err.filtered-081217 14:24:09  InnoDB: Database was not shut down normally.
/var/log/mysql.err.filtered-InnoDB: Starting recovery from log files...
--
/var/log/mysql.err.filtered:Stack range sanity check OK, backtrace follows:
/var/log/mysql.err.filtered-0x80fa10b ha_heap::rnd_next(char*) + 11
/var/log/mysql.err.filtered-0xffffe400 _end + -138282244
/var/log/mysql.err.filtered-0xa3dfe5dc _end + -1683882792
/var/log/mysql.err.filtered-0x82a0b86 wcrtomb + 150
/var/log/mysql.err.filtered-0x82a6684 __register_atfork + 152
/var/log/mysql.err.filtered-0x82a6422 clone + 98
/var/log/mysql.err.filtered-0x80f0704 and_all_keys(SEL_ARG*, SEL_ARG*, unsigned int) + 292
/var/log/mysql.err.filtered-0x80fb883 ha_myisam::repair(THD*, st_mi_check_param&, bool) + 259
/var/log/mysql.err.filtered-0x80fb234 ha_myisam::restore(THD*, st_ha_check_opt*) + 116
/var/log/mysql.err.filtered-0xb7d90ea8 _end + -1348786780
/var/log/mysql.err.filtered-0x80b5d19 close_temporary_tables(THD*) + 409
/var/log/mysql.err.filtered-New value of fp=(nil) failed sanity check, terminating stack trace!
/var/log/mysql.err.filtered-Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
/var/log/mysql.err.filtered-stack trace is much more helpful in diagnosing the problem, so please do
/var/log/mysql.err.filtered-resolve it
/var/log/mysql.err.filtered-The manual page at http://www.mysql.com/doc/en/Crashing.html contains
/var/log/mysql.err.filtered-information that should help you find out what is causing the crash.

Hardware: 
             total       used       free     shared    buffers     cached
Mem:       8317408    8028164     289244          0       1744    7611852
-/+ buffers/cache:     414568    7902840
Swap:       417672          8     417664

Linux xxxxx 2.6.25.4-i386 #1 SMP Tue May 20 14:41:01 CEST 2008 i686 GNU/Linux

How to repeat:
unknown, happens from time to time.

Suggested fix:
none known

[17 Dec 2008 14:57] Valeriy Kravchuk

Thank you for a problem report. Please, send your my.cnf file content. Had you identifed the exact SQL statement that leads to crash?

[17 Dec 2008 20:17] Robert Heinzmann

Here's the config, without comments
# cat /etc/my.cnf
[debian]
debian_error_log        = /var/log/mysql.err

[safe_mysqld]
err-log = /var/log/mysql.err
open-files=8192

[client]
port            = 3306
socket          = /var/run/mysqld/mysqld.sock

[mysqld]
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
log-err         = /var/log/mysql.err
log-slow-queries= /var/log/mysql.slowlog
log-warnings
datadir         = /db/mysql
tmpdir          = /var/tmp
language        = english
myisam-recover  = BACKUP,FORCE
skip-locking
skip-bdb
safe-user-create
safe-show-database
key_buffer=64M
max_allowed_packet=10M
table_cache=64
sort_buffer=512K
myisam_sort_buffer_size=8M
max_connections=240
max_connect_errors=1000
wait_timeout=60
interactive_timeout=3600
flush_time=300
long_query_time=3
open_files_limit=768
default_week_format=3
query_cache_type = 1
query_cache_size = 32M
innodb_data_home_dir =
innodb_data_file_path = /db/mysql/ibdata1:10M:autoextend
innodb_additional_mem_pool_size=20M
innodb_buffer_pool_size=128M
innodb_lock_wait_timeout=60
thread_cache_size=20

[mysqld-4.0]
default-character-set=german1
basedir         = /usr/

[mysqld-4.1]
collation-server=latin1_german2_ci
old-passwords
basedir         = /usr/local/mysql

[mysqld-5.0]
collation-server=latin1_german2_ci
old-passwords
basedir         = /usr/local/mysql5

[mysqldump]
quick
set-variable    = max_allowed_packet=10M
set-variable    = net_buffer_length=512K
quote-names
allow-keywords

[mysql]
no-auto-rehash

[isamchk]
set-variable    = key_buffer=20M
set-variable    = sort_buffer=20M
set-variable    = read_buffer=2M
set-variable    = write_buffer=2M

[myisamchk]
set-variable    = key_buffer=20M
set-variable    = sort_buffer=20M
set-variable    = read_buffer=2M
set-variable    = write_buffer=2M
force
tmpdir=/db/check

[mysqlcheck]
force
auto-repair

[mysqladmin]
connect_timeout=20
shutdown_timeout=600

--------

There is no exact query to identify, sorry.

[19 Dec 2008 12:10] Robert Heinzmann

Problem could be identified. There is one database and if we dump that database, the server reproducable crashes. We have a binary dump of that database, however I can not send the database (privacy). 

How can I debug the issue ? Valgrind ? 
http://forge.mysql.com/wiki/Checking_Memory_With_Valgrind

[22 Dec 2008 14:05] MySQL Verification Team

Thank you for the feedback. You can send the dump file privacy at:

ftp://ftp.mysql.com/pub/mysql/upload

use a file name which identifies this bug report and comment here when done.
Thanks in advance.

[22 Dec 2008 15:24] Robert Heinzmann

Uploaded the valgrind dump with:

  mysqld_crash_bug_41549.txt.2022

The dump was generated with 
mysqldump --defaults-file=/root/sandboxes/msb_4_0_27/my.sandbox.cnf db171452465 --all --force > /dev/null

Where db171452465 is the database reproducably crashing the host.

[12 Jan 2009 12:00] MySQL Verification Team

Hi Robert - valgrind output indeed shows invalid reads and writes of memory.
Next steps would be:

o) run check table on each table in the database - repair any corrupted ones
o) try using non-icc binaries and see if valgrind still complains.

[13 Feb 2009 0:00] Bugs System

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".