Bug #29463 server died, falcon recovery fails in loop, crashing over and over
Submitted: 1 Jul 2007 1:11 Modified: 6 Jul 2007 15:49
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:6.0.1-BK OS:Linux (suse 9.3)
Assigned to: CPU Architecture:Any
Tags: covery, crash

[1 Jul 2007 1:11] Shane Bester
Description:
place the dump.txt into /tmp/ and run the sql commands in the uploaded file. this test is designed to crash at "load data ... into tbl_18".

after that crash, server happily continues, until some time later, where it's crashing over and over.

I had to upload stack traces in a file because there so many crashes.

070701  2:44:54 - mysqld got signal 4;
0x820fa3b handle_segfault + 541
0xffffe410 _end + -142788864
0x83e4782 _ZN5Error10debugBreakEv + 18
0x83e46fa _ZN5Error5errorEPKcz + 82
0x83e4768 _ZN5Error15assertionFailedEPKci + 32
0x83f2b4b _ZN13IndexRootPage10indexMergeEP3DbbiP14SRLUpdateIndexj + 2363
0x8432d3f _ZN14SRLUpdateIndex7executeEv + 163
0x8432c61 _ZN14SRLUpdateIndex6commitEv + 99
0x8421202 _ZN20SerialLogTransaction6commitEv + 222
0x84213e9 _ZN20SerialLogTransaction8doActionEv + 25
0x841d908 _ZN9SerialLog12gopherThreadEv + 206
0x841d837 _ZN9SerialLog12gopherThreadEPv + 17
0x83bd547 _ZN6Thread6threadEv + 67
0x83bd4e6 _ZN6Thread6threadEPv + 18
Bugcheck: assertion failed at line 1028 in file IndexRootPage.cpp

0x820fa3b handle_segfault + 541
0xffffe410 _end + -142788864
0x83e4782 _ZN5Error10debugBreakEv + 18
0x83e46fa _ZN5Error5errorEPKcz + 82
0x83e4768 _ZN5Error15assertionFailedEPKci + 32
0x83f2b4b _ZN13IndexRootPage10indexMergeEP3DbbiP14SRLUpdateIndexj + 2363
0x8432d3f _ZN14SRLUpdateIndex7executeEv + 163
0x8432bfb _ZN14SRLUpdateIndex4redoEv + 17
0x841e1a4 _ZN9SerialLog7recoverEv + 1628
0x83d7642 _ZN8Database12openDatabaseEPKc + 756
0x83d3544 _ZN10Connection11getDatabaseEPKcS1_P7Threads + 256
0x83d1ad5 _ZN10Connection12openDatabaseEPKcS1_S1_S1_S1_P7Threads + 237
0x83ac29d _ZN15StorageDatabase17getOpenConnectionEv + 113
0x83affdd _ZN14StorageHandler10initializeEv + 131
0x83af637 _ZN14StorageHandler20getStorageConnectionEP17StorageTableShareP3THDi10OpenOptioni + 75

0x820fa3b handle_segfault + 541
0xffffe410 _end + -142788864
0x83e4782 _ZN5Error10debugBreakEv + 18
0x83e46fa _ZN5Error5errorEPKcz + 82
0x84469b0 _ZN5Cache9fetchPageEP3Dbbi8PageType8LockType + 592
0x83dd7f7 _ZN3Dbb9fetchPageEi8PageType8LockType + 51
0x8417090 _ZN7Section12updateRecordEiP6Streamjb + 496
0x83ddc03 _ZN3Dbb12updateRecordEiiP6Streamjb + 73
0x8433c78 _ZN16SRLUpdateRecords4redoEv + 444
0x841e1a4 _ZN9SerialLog7recoverEv + 1628
0x83d7642 _ZN8Database12openDatabaseEPKc + 756
0x83d3544 _ZN10Connection11getDatabaseEPKcS1_P7Threads + 256
0x83d1ad5 _ZN10Connection12openDatabaseEPKcS1_S1_S1_S1_P7Threads + 237
0x83ac29d _ZN15StorageDatabase17getOpenConnectionEv + 113
0x83affdd _ZN14StorageHandler10initializeEv + 131
0x83af637 _ZN14StorageHandler20getStorageConnectionEP17StorageTableShareP3THDi10OpenOptioni + 75
0x83a4ac4 _ZN16StorageInterface4openEPKcij + 202
0x82f997d _ZN7handler7ha_openEP8st_tablePKcii + 261
0x82630a3 _Z21open_table_from_shareP3THDP14st_table_sharePKcjjjP8st_tableb + 2055
0x82584ba _Z17open_unireg_entryP3THDP8st_tableP13st_table_listPKcPcjP11st_mem_rootj + 1502
0x8256a27 _Z10open_tableP3THDP13st_table_listP11st_mem_rootPbj + 3139

<....>

How to repeat:
start with a clean datadir.
place dump.txt into /tmp
run sql commands in the other text file.

Suggested fix:
fix falcon's recovery, or whatever is causing a crash over and over.
[1 Jul 2007 1:13] MySQL Verification Team
dump file for static data.

Attachment: dump.txt (text/plain), 6.31 KiB.

[1 Jul 2007 1:14] MySQL Verification Team
run all these sql statements, making sure mysqld_safe will restart server if it crashes.

Attachment: bug29463_error_log.txt.bz2 (application/octet-stream, text), 16.53 KiB.

[1 Jul 2007 1:14] MySQL Verification Team
error logs from my crashes

Attachment: bug29463_error_log.txt.bz2 (application/octet-stream, text), 16.53 KiB.

[1 Jul 2007 1:15] MySQL Verification Team
run all *these* sql statements, making sure mysqld_safe will restart server if it crashes.

Attachment: bug29463_testcase.txt.bz2 (application/octet-stream, text), 12.07 KiB.

[1 Jul 2007 13:16] Hakan Küçükyılmaz
Verified as described. Crash at tbl_18

hakan@lu0011:~/work/mysql/falcon$ mysqladmin -uroot  create test
hakan@lu0011:~/work/mysql/falcon$ mysql -uroot test < bug29463_testcase.txt
Table   Op      Msg_type        Msg_text
test.tbl_0      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_1      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_2      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_3      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_4      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_5      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_6      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_7      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_8      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_9      check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_10     check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_11     check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_12     check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_13     check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_14     check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_15     check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_16     check   note    The storage engine for the table doesn't support check
Table   Op      Msg_type        Msg_text
test.tbl_17     check   note    The storage engine for the table doesn't support check

--> crash

Backtrace is:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1074272576 (LWP 5810)]
0x0000000000964984 in my_strnxfrm_czech (cs=<value optimized out>,
    dst=0x4007e3d8 "M\001 \001\003\001\003\003\001��", dstlen=71, nweights_arg=1100, src=0x4007dec0 "�\024",
    srclen=<value optimized out>, flags=15) at ctype-czech.c:464
464             NEXT_CMP_VALUE(src, p, level, value, (int) srclen, 0);
Current language:  auto; currently c

(gdb) bt
#0  0x0000000000964984 in my_strnxfrm_czech (cs=<value optimized out>,
    dst=0x4007e3d8 "M\001 \001\003\001\003\003\001��", dstlen=71, nweights_arg=1100, src=0x4007dec0 "�\024",
    srclen=<value optimized out>, flags=15) at ctype-czech.c:464
#1  0x00000000007bf88e in falcon_strnxfrm (cs=0xff,
    dst=0x73635f6863657a <Address 0x73635f6863657a out of bounds>, dstlen=71, src=0x4007dec0 "�\024", srclen=2)
    at ha_falcon.cpp:179
#2  0x0000000000810167 in MySQLCollation::makeKey (this=0x2aaaac6883d8, value=<value optimized out>,
    key=0x4007e3c0, partialKey=0) at MySQLCollation.cpp:73
#3  0x0000000000805114 in Index::makeKey (this=<value optimized out>, field=0x2aaaac687d08, value=0x4007e950,
    segment=0, indexKey=0x4007e3c0) at Index.cpp:288
#4  0x000000000080533b in Index::makeKey (this=0x2aaaac6beb30, count=2, values=0x4007ed50, indexKey=0x4007ee90)
    at Index.cpp:355
#5  0x000000000080564d in Index::makeKey (this=0x2aaaac6beb30, record=0x2aaaac6f1e10, key=0x4007ee90)
    at Index.cpp:535
#6  0x0000000000805a95 in Index::insert (this=0x2aaaac6beb30, record=0x2aaaac6f1e10,
    transaction=0x2b6ed4cc7f20) at Index.cpp:189
#7  0x00000000007d42ea in Table::insert (this=0x2aaaac6bdc00, transaction=0x2b6ed4cc7f20,
    stream=0x2b6ed4a9a1e8) at Table.cpp:2553
#8  0x00000000007c7f0b in StorageDatabase::insert (this=<value optimized out>,
    connection=<value optimized out>, table=0x2aaaac6bdc00, stream=0x2b6ed4a9a1e8) at StorageDatabase.cpp:215
#9  0x00000000007cb848 in StorageTable::insert (this=0x2b6ed4a99670) at StorageTable.cpp:88
#10 0x00000000007c54bc in StorageInterface::write_row (this=0xff9ba8,
    buff=0xff9de0 "�\002�\024\002�\024���������\b\236�") at ha_falcon.cpp:887
#11 0x000000000071963f in handler::ha_write_row (this=0xff,
    buf=0x73635f6863657a <Address 0x73635f6863657a out of bounds>) at handler.cc:3669
#12 0x00000000006b235c in write_record (thd=0xf74618, table=0xff86e8, info=0x4007fb20) at sql_insert.cc:1315
#13 0x000000000073b422 in mysql_load (thd=0xf74618, ex=0xfcb5c8, table_list=0xfce590, fields_vars=@0xf764e0,
    set_fields=@0xf76510, set_values=@0xf764f8, handle_duplicates=DUP_REPLACE, ignore=false,
    read_file_from_client=true) at sql_load.cc:822
#14 0x0000000000646f42 in mysql_execute_command (thd=0xf74618) at sql_parse.cc:2983
#15 0x000000000064ab9a in mysql_parse (thd=0xf74618,
    inBuf=0xfce428 "load data local infile '/tmp/dump.txt' replace into table tbl_18 (@h,@i) set a=UNHEX(@h),b=UNHEX(@h)", length=100, found_semicolon=0x40080ff0) at sql_parse.cc:5381
#16 0x000000000064b82c in dispatch_command (command=COM_QUERY, thd=0xf74618, packet=<value optimized out>,
    packet_length=101) at sql_parse.cc:907
#17 0x000000000064c6e0 in do_command (thd=0xf74618) at sql_parse.cc:669
#18 0x000000000063e4b4 in handle_one_connection (arg=<value optimized out>) at sql_connect.cc:1091
#19 0x00002b6ed33a0225 in start_thread () from /lib/libpthread.so.0
#20 0x00002b6ed416c17d in clone () from /lib/libc.so.6
[1 Jul 2007 13:25] Hakan Küçükyılmaz
We are hitting the crash when running the tbl_18 part only, too

hakan@lu0011:~/work/mysql/falcon$ mysql -uroot test
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 6.0.1-alpha-debug Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

6.0.1-alpha-debug
[15:24] root@test>create table tbl_18 (a varchar(2) character set latin2 collate latin2_czech_cs,b varchar(2) character set latin2 collate latin2_czech_cs,key(a,b),key(b,a),key(b),key(a))engine=falcon;
Query OK, 0 rows affected (0.24 sec)

6.0.1-alpha-debug
[15:24] root@test>load data local infile '/tmp/dump.txt' replace into table tbl_18 (@h,@i) set a=UNHEX(@h),b=UNHEX(@h);

---> crash
[1 Jul 2007 13:59] MySQL Verification Team
Please note that this testcase will not work after fixing bug #29459
This is because there will be no crash and therefore no failed crash recovery.
[6 Jul 2007 15:49] Jim Starkey
The script runs to completion on both WinXP and AMD64 Linux.