Bug #44606 Crash on deletion of an InnoDB-Table
Submitted: 1 May 2009 14:52 Modified: 24 Sep 2009 9:07
Reporter: Michael Diekmann Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.0.67 OS:Linux (openSUSE 11.0 (i586))
Assigned to: CPU Architecture:Any
Tags: crash, innodb

[1 May 2009 14:52] Michael Diekmann
Description:
Linux 2.6.25.18-0.2-pae (gcc 4.3.1 20080507)

Compiled from source (mysql-5.0.67.tar.gz) with:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./configure \
--prefix=/usr/local/mysql/5.0.67 \
--localstatedir=/sql/data \
--with-mysqld-user=mysql \
--with-unix-socket-path=/sql/mysql.sock \
--enable-assembler \
--with-zlib-dir=/usr/local/ \
--with-openssl=/usr/local/openssl/current/ \
--without-debug \
--without-bench \
--without-docs 

The symbols can be found here: http://www.undef.de/.Download/symbols_mysql_5.0.67_crash.txt

I just have started a deletion of a table (named 'DBNAME'), the MySQL-Server did not complete this deletion and crashes. See log below.

I have changed some database- and tablenames in the logfile below:
 - DBNAME and DBNAME1 are different databases
 - TABLENAME and TABLENAME1 are also different

The logfile contains some german error messages, because i'am german. ;)

Below the logfile, i have a copy of the "my.cnf".

---- cut here logfile ---
090118 14:01:32  mysqld started
090118 14:01:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
090118 14:01:32 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
090118 14:01:33  InnoDB: Started; log sequence number 40 423332417
090118 14:01:34 [Note] /usr/local/mysql/current/libexec/mysqld: bereit für Verbindungen.
Version: '5.0.67'  Socket: '/sql/mysql.sock'  Port: 3306  Source distribution
090413 16:30:19 [ERROR] /usr/local/mysql/current/libexec/mysqld: Tabelle './DBNAME/TABLENAME' ist als defekt markiert und sollte repariert werden
090501 13:45:23 [ERROR] /usr/local/mysql/current/libexec/mysqld: Tabelle './DBNAME1/TABLENAME1' ist als defekt markiert und sollte repariert werden
090501 13:45:23 [ERROR] /usr/local/mysql/current/libexec/mysqld: Tabelle './DBNAME1/TABLENAME1' ist als defekt markiert und sollte repariert werden
090501 15:58:18InnoDB: Assertion failure in thread 2221530000 in file row0mysql.c line 3340
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
090501 15:58:18 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=57
max_connections=100
threads_connected=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1056768 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x958c958
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x84699e18, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81a6c36
0x831fe05
0x8285ecc
0x8272389
0x8291b7e
0x8291e20
0x8286943
0x8287d6c
0x81c1a18
0x81c6d25
0x81c8248
0x81c9855
0xb7dc2175
0xb7bd5dce
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x998ecb8 = DROP DATABASE `DBNAME`
thd->thread_id=2014869
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

The "--memlock" argument, which was enabled, uses system calls that are
unreliable and unstable on some operating systems and operating-system
versions (notably, some versions of Linux).  This crash could be due to use
of those buggy OS calls.  You should consider whether you really need the
"--memlock" parameter and/or consult the OS distributer about "mlockall"
bugs.

Number of processes running now: 0
090501 15:58:20  mysqld restarted
090501 15:58:21 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
090501 15:58:21 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
090501 15:58:21  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
090501 15:58:23  InnoDB: Started; log sequence number 112 2704498107
090501 15:58:24 [ERROR] /usr/local/mysql/current/libexec/mysqld: Tabelle './mysql/user' ist als defekt markiert und sollte repariert werden
090501 15:58:24 [Warning] Checking table:   './mysql/user'
--- cut here logfile ---

The last two line comes for all MyISAM-Tables.

--- my.cnf ---
[mysqld]
server-id			= 1
language			= german
default_character_set		= utf8
user				= mysql
port				= 3306
socket				= /sql/mysql.sock
datadir				= /sql/data
tmpdir				= /backup/tmp/mysql/

back_log			= 50
max_connections			= 100
max_connect_errors		= 10

key_buffer			= 384M
max_allowed_packet		= 5M
binlog_cache_size		= 1M
max_heap_table_size		= 64M
sort_buffer_size		= 8M
join_buffer_size		= 8M
thread_cache_size		= 8
thread_concurrency		= 8
query_cache_size		= 64M
query_cache_limit		= 2M
ft_min_word_len			= 3
tmp_table_size			= 64M
table_cache			= 2048
default_table_type		= MYISAM

memlock									
safe-show-database
skip-symlink
skip-locking
skip-name-resolve
skip-federated

# *** MyISAM Specific options ***
key_buffer_size			= 32M
read_buffer_size		= 2M
read_rnd_buffer_size		= 16M
bulk_insert_buffer_size		= 64M
myisam_sort_buffer_size		= 128M
myisam_max_sort_file_size	= 5G
myisam_max_extra_sort_file_size	= 5G
myisam_repair_threads		= 1
myisam_recover

# *** INNODB Specific options ***
innodb_data_home_dir		= /sql/data/
innodb_data_file_path		= ibdata1:10M:autoextend:max:1990M
innodb_additional_mem_pool_size	= 32M
innodb_buffer_pool_size		= 512M
innodb_file_io_threads		= 4
innodb_thread_concurrency	= 8
innodb_flush_log_at_trx_commit	= 1
innodb_log_buffer_size		= 64M
innodb_log_file_size		= 256M
innodb_log_files_in_group	= 3
innodb_max_dirty_pages_pct	= 50
innodb_lock_wait_timeout	= 60
innodb_open_files		= 1000
innodb_fast_shutdown
innodb_file_per_table
--- my.cnf ---

How to repeat:
Sorry, i did'nt know how to repeat this.
[3 May 2009 12:10] Valeriy Kravchuk
Thank you for the problem report. Please, send resolved stack trace for this crash. The results of ls -l in the database directory may be also useful.
[3 May 2009 17:16] Michael Diekmann
Stack trace

Attachment: symbols_mysql_5.0.67_crash.txt (text/plain), 25.72 KiB.

[3 Jun 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[4 Jun 2009 18:34] Michael Diekmann
This Bug-Report is still open.
[26 Jul 2009 9:39] Valeriy Kravchuk
If you still use 5.0.67 on the same box, please, resolve stack trace from the crash:

0x81a6c36
0x831fe05
0x8285ecc
0x8272389
0x8291b7e
0x8291e20
0x8286943
0x8287d6c
0x81c1a18
0x81c6d25
0x81c8248
0x81c9855
0xb7dc2175
0xb7bd5dce

as described at http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and send the results.
[27 Jul 2009 18:19] Michael Diekmann
stack_trace_mysql_5.0.67_crash.txt

Attachment: stack_trace_mysql_5.0.67_crash.txt (text/plain), 780 bytes.

[6 Sep 2009 19:11] Valeriy Kravchuk
Please, check if this crash is repeatable with memlock option commented out in my.cnf.
[7 Sep 2009 18:21] Michael Diekmann
Sorry, but this is not repeatable. The structur of database was changed and the bug has reached. So it is impossible to repeat this.
[24 Sep 2009 9:07] Susanne Ebrecht
I will set this to can't repeat.