Bug #31317 falcon crash recovery not perfect yet
Submitted: 1 Oct 2007 15:13 Modified: 19 Nov 2007 16:14
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S3 (Non-critical)
Version:6.0.4-alpha-debug OS:Any
Assigned to: CPU Architecture:Any
Tags: crash, crash recovery

[1 Oct 2007 15:13] Shane Bester
Description:
I built today's BK sources from bkf clone bk://mysql.bkbits.net/mysql-6.0-falcon

crashing mysqld many times eventually led to an unusable table(space?) and now I cannot even drop the table.  First falcon crash looking like this:

Version: '6.0.4-alpha-debug'  socket: '/tmp/mysql.sock'  port: 3306  yes
071001  2:24:11 - mysqld got signal 4;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388572
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337629 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x8e9df90
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x42d0a19c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x822136b handle_segfault + 541
0xffffe410 _end + -143089920
0x840c572 Error::debugBreak() + 18
0x840c4ea Error::error(char const*, ...) + 82
0x8470b59 Cache::fetchPage(Dbb*, int, PageType, LockType) + 579
0x8405385 Dbb::fetchPage(int, PageType, LockType) + 51
0x8418755 IndexRootPage::findRoot(Dbb*, int, int, LockType, unsigned int) + 55
0x841837c IndexRootPage::findLeaf(Dbb*, int, int, IndexKey*, LockType, unsigned int) + 48
0x84188e0 IndexRootPage::scanIndex(Dbb*, int, int, IndexKey*, IndexKey*, int, unsigned int, Bitmap*) + 232
0x8416801 Index::scanIndex(IndexKey*, IndexKey*, int, Transaction*, Bitmap*) + 545
0x83e1a86 Table::checkUniqueIndexes(Transaction*, RecordVersion*) + 184
0x83e2af8 Table::insert(Transaction*, Stream*) + 234
0x83d2ef9 StorageDatabase::insert(Connection*, Table*, Stream*) + 39
0x83d75b7 StorageTable::insert() + 59
0x83cc6d7 StorageInterface::write_row(unsigned char*) + 253
0x831d72f handler::ha_write_row(unsigned char*) + 33
0x82a7c5d write_record(THD*, st_table*, st_copy_info*) + 1969
0x82ab6be select_insert::send_data(List<Item>&) + 366
0x82992c2 end_send(JOIN*, st_join_table*, bool) + 212
0x8297753 evaluate_join_record(JOIN*, st_join_table*, int, char*) + 539
0x82972f8 sub_select(JOIN*, st_join_table*, bool) + 232
0x8296e28 do_select(JOIN*, List<Item>*, st_table*, Procedure*) + 756
0x82823bb JOIN::exec() + 7201
0x8282952 _Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select + 694
0x827c3ce handle_select(THD*, st_lex*, select_result*, unsigned long) + 378
0x822fed8 mysql_execute_command(THD*) + 10078
0x8236970 mysql_parse(THD*, char const*, unsigned int, char const**) + 372
0x822c1ec dispatch_command(enum_server_command, THD*, char*, unsigned int) + 2128
0x822b990 do_command(THD*) + 600
0x822a5c5 handle_one_connection + 255
0x40038aa7 _end + 930891159
0x401f8c2e _end + 932726558
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
and follow instructions on how to resolve the stack trace.
Resolved stack trace is much more helpful in diagnosing the
problem, so please do resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8ecdc98 = insert into t1(col000) select (convert(`t1`.`a`,datetime)) from t1 where col000 <> (least(`t1`.`col000`,`t1`.`a`,`t1`.`a`,`t1`.`col001`)) limit 1
thd->thread_id=1

subsequent attempts to drop the table had same crashes. I had some 20MB of the same crashes in my logs.

How to repeat:
run some testcases over and over for known crashing bugs, and crash mysqld repeatedly, having mysqld_safe restart it each time.  Some such bugs might be:

bug #31253
bug #31159
bug #31156
bug #31152

etc.

Suggested fix:
improve crash recovery code.
[1 Oct 2007 15:15] MySQL Verification Team
I forgot to mention, obviously you must say "engine=falcon" for those known crash bug's testcases :)
[1 Oct 2007 16:43] MySQL Verification Team
was at least 5 hours of testing/crashing before the error occurred.  I'll try make a testcase, but no promises for it soon.
[16 Oct 2007 11:26] Hakan Küçükyılmaz
Shane,

can you get a reproducable way to verify this bug?

Best regards,

Hakan
[19 Oct 2007 16:14] Kevin Lewis
Shane, if this happens again for you, please send us the logs and tablespaces so that we can reproduce it.
[20 Nov 2007 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".