Description:
When attemptimg to drop a database consisting of about 50 InnoDB tables, the server repeatedly crashed, the database seemed to survive intact. The issue was resolved by removal of both the database directory and the ib_logfile[01] journals.
How to repeat:
Unfortunately, this has happened only once. Also there was no core file we could send, but I did capture an innodb_monitor log of it happenning. Sorry about losing the datafiles and journals. if it happens again I'll ftp them over. I'm sure the root cause was corrupted files since this was repeatable until I nuked the db.
060217 15:38:17 InnoDB: Started; log sequence number 0 378619946
060217 15:38:17 [Note] /work/mysql/bin/mysqld: ready for connections.
Version: '5.0.15-standard' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL)
=====================================
060217 15:39:05 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 48 seconds
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 3, signal count 3
Mutex spin waits 1, rounds 20, OS waits 0
RW-shared spins 6, OS waits 3; RW-excl spins 1, OS waits 0
------------
TRANSACTIONS
------------
Trx id counter 0 494341
Purge done for trx's n:o < 0 494341 undo n:o < 0 0
History list length 10
Total number of lock structs in row lock hash table 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0 0, not started, process no 20454, OS thread id 1116154800
MySQL thread id 1, query id 80 localhost root
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (write thread)
Pending normal aio reads: 0, aio writes: 0,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
163 OS file reads, 21 OS file writes, 11 OS fsyncs
3.40 reads/s, 30581 avg bytes/read, 0.44 writes/s, 0.23 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf for space 0: size 1, free list len 0, seg size 2, is empty
Ibuf for space 0: size 1, free list len 0, seg size 2,
0 inserts, 0 merged recs, 0 merges
Hash table size 34679, used cells 142, node heap has 1 buffer(s)
0.10 hash searches/s, 10.23 non-hash searches/s
---
LOG
---
Log sequence number 0 378622401
Log flushed up to 0 378622401
Last checkpoint at 0 378622401
0 pending log writes, 0 pending chkp writes
12 log i/o's done, 0.25 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 18720102; in additional pool allocated 1002496
Buffer pool size 512
Free buffers 341
Database pages 170
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages read 170, created 0, written 18
3.54 reads/s, 0.00 creates/s, 0.37 writes/s
Buffer pool hit rate 962 / 1000
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
1 read views open inside InnoDB
Main thread process no. 20454, id 1107233712, state: waiting for server activity
Number of rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
InnoDB: Dump of the tablespace extent descriptor: len 40; hex 00000000000000000000000001b6ffffffff000000000002abaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa; asc ;
InnoDB: Serious error! InnoDB is trying to free page 0
InnoDB: though it is already marked as free in the tablespace!
InnoDB: The tablespace free space info is corrupt.
InnoDB: You may need to dump your InnoDB tables and recreate the whole
InnoDB: database!
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html
InnoDB: about forcing recovery.
060217 15:39:11InnoDB: Assertion failure in thread 1116154800 in file fsp0fsp.c line 2980
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html
InnoDB: about forcing recovery.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x89fc6a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x4286ad0c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x814e7f8
0xffffe420
0x82931bd
0x82931bd
0x824d53a
0x8283c52
0x82832d8
0x8282706
0x8261aa4
0x8260f55
0x827432d
0x81f3d97
0x81e8c69
0x81f9f81
0x81f998e
0x81f92a7
Stack trace seems successful - bottom reached
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8a28718 = drop database smsdb
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
0x814e7f8 handle_segfault + 356
0xffffe420 _end + -139683376
0x82931bd btr_free_but_not_root + 237
0x82931bd btr_free_but_not_root + 237
0x824d53a dict_drop_index_tree + 142
0x8283c52 row_upd_clust_step + 1294
0x82832d8 row_upd + 404
0x8282706 row_upd_step + 198
0x8261aa4 que_thr_step + 920
0x8260f55 que_run_threads + 45
0x827432d row_drop_table_for_mysql + 1897
0x81f3d97 _ZN11ha_innobase12delete_tableEPKc + 283
0x81e8c69 _Z15ha_delete_tableP3THD7db_typePKcS3_b + 133
0x81f9f81 _Z20mysql_rm_table_part2P3THDP13st_table_listbbbb + 1429
0x81f998e _Z30mysql_rm_table_part2_with_lockP3THDP13st_table_listbbb + 90
0x81f92a7 _Z20mysql_rm_known_filesP3THDP9st_my_dirPKcS4_jPP13st_table_list + 731
Suggested fix:
Don't let it happen again :-). It's tough to suggest any course of action, but perhaps you can get something from the log
