Bug #41472 Falcon crash in MemFreeBlock::findNextLargest
Submitted: 15 Dec 2008 13:54 Modified: 26 May 2010 17:50
Reporter: Philip Stoev Email Updates:
Status: Unsupported Impact on me:
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:6.0-falcon-team OS:Any
Assigned to: Kevin Lewis CPU Architecture:Any

[15 Dec 2008 13:54] Philip Stoev
When executing a random transactional workload, Falcon crashed as follows:

#2  0x00000000006ba74c in handle_segfault (sig=11) at mysqld.cc:2658
#3  <signal handler called>
#4  0x0000000000a7e56f in MemFreeBlock::findNextLargest (this=0x153d9e8, size=72) at MemFreeBlock.cpp:153
#5  0x00000000009ed28d in MemMgr::alloc (this=0x153d9a0, s=56) at MemMgr.cpp:431
#6  0x00000000009ed686 in MemMgr::allocateDebug (this=0x153d9a0, size=17, fileName=0xe14d08 "Record.cpp", line=921) at MemMgr.cpp:559
#7  0x00000000009eea5d in MemMgrPoolAllocateDebug (pool=0x153d9a0, s=17, file=0xe14d08 "Record.cpp", line=921) at MemMgr.cpp:117
#8  0x0000000000a08ec2 in operator new [] () at MemoryManager.h:74
#9  Record::allocRecordData (this=0x7fdb076a2d50, length=17) at Record.cpp:921
#10 0x0000000000a0aa8c in Record::setEncodedRecord (this=0x7fdb076a2d50, stream=0x7fdb0aafe458, interlocked=false) at Record.cpp:707
#11 0x000000000097d4e2 in Table::insert (this=0x7fdb12728b28, transaction=0x2b5bf58, stream=0x7fdb0aafe458) at Table.cpp:3040
#12 0x0000000000961419 in StorageDatabase::insert (this=0x7fdb12337210, connection=0x7fdb12383548, table=0x7fdb12728b28, stream=0x7fdb0aafe458)
    at StorageDatabase.cpp:266
#13 0x00000000009685c3 in StorageTable::insert (this=0x7fdb0aaf8eb8) at StorageTable.cpp:109
#14 0x000000000095a999 in StorageInterface::write_row (this=0x2a834f0, buff=0x2a837b0 "ЫY\001") at ha_falcon.cpp:1132
#15 0x00000000008005e7 in handler::ha_write_row (this=0x2a834f0, buf=0x2a837b0 "ЫY\001") at handler.cc:5369
#16 0x0000000000768ea3 in write_record (thd=0x7fdb0c92f428, table=0x2a5b4a8, info=0x7fdaff8b94f0) at sql_insert.cc:1382
#17 0x000000000076d42d in mysql_insert (thd=0x7fdb0c92f428, table_list=0x2a42de0, fields=@0x7fdb0c931908, values_list=@0x7fdb0c931950,
    update_fields=@0x7fdb0c931938, update_values=@0x7fdb0c931920, duplic=DUP_REPLACE, ignore=false) at sql_insert.cc:835
#18 0x00000000006ce105 in mysql_execute_command (thd=0x7fdb0c92f428) at sql_parse.cc:3110
#19 0x00000000006d37d1 in mysql_parse (thd=0x7fdb0c92f428,
    inBuf=0x2a42bd0 "REPLACE INTO `table10_falcon_int_autoinc` ( `pk` , `int_key` , `int` ) VALUES ( `pk` , 345 , 567 )", length=98,
    found_semicolon=0x7fdaff8baf00) at sql_parse.cc:5732
#20 0x00000000006d43bc in dispatch_command (command=COM_QUERY, thd=0x7fdb0c92f428,
    packet=0x7fdb0c931f39 " REPLACE INTO `table10_falcon_int_autoinc` ( `pk` , `int_key` , `int` ) VALUES ( `pk` , 345 , 567 ) ", packet_length=100)
    at sql_parse.cc:1007
#21 0x00000000006d58e5 in do_command (thd=0x7fdb0c92f428) at sql_parse.cc:690
#22 0x00000000006c3a69 in handle_one_connection (arg=0x7fdb0c92f428) at sql_connect.cc:1154
#23 0x000000315b0073da in start_thread () from /lib64/libpthread.so.0
#24 0x000000315a4e627d in clone () from /lib64/libc.so.6

(gdb) list
148             MemFreeBlock *block = this;
150             // Travse down the tree looking for a block that fits
152             while (block)
153                     if (size < block->memHeader.length)
154                             {
155                             if (block->smaller)
156                                     block = block->smaller;
157                             else

(gdb) print block
$1 = (MemFreeBlock *) 0x7fdb076a3f
(gdb) print block->memHeader
Cannot access memory at address 0x7fdb076a4f

How to repeat:
If this is repeatable, a test case will be provided shortly.
[13 May 2009 10:57] Olav Sandstå
The same crash happened when running the falcon_chill_thaw test using the latest source from the mysql-6.0-falcon-team tree.

The call stack is identical for the parts that involves the Falcon memory manager but it is called from a different part of the Falcon code. The new call stack:

#3  0x082c2082 in handle_segfault (sig=11) at mysqld.cc:2710
#4  <signal handler called>
#5  0x085e2d15 in MemFreeBlock::findNextLargest (this=0x8becf74, size=64)
    at MemFreeBlock.cpp:153
#6  0x0856533c in MemMgr::alloc (this=0x8becf40, s=64) at MemMgr.cpp:453
#7  0x08565687 in MemMgr::allocateDebug (this=0x8becf40, size=35, 
    fileName=0x8a01160 "Record.cpp", line=1043) at MemMgr.cpp:581
#8  0x085669a3 in MemMgrPoolAllocateDebug (pool=0x8becf40, s=35, 
    file=0x8a01160 "Record.cpp", line=1043) at MemMgr.cpp:126
#9  0x085f91c1 in Record::allocRecordData (this=0xae326fe8, length=35)
    at MemoryManager.h:75
#10 0x085f95fc in Record::setEncodedRecord (this=0xae326fe8, 
    stream=0xa707f284, interlocked=true) at Record.cpp:728
#11 0x085fde62 in RecordVersion::thaw (this=0xae326fe8)
    at RecordVersion.cpp:436
#12 0x085fdb4f in RecordVersion::getRecordData (this=0xae326fe8)
    at RecordVersion.cpp:538
#13 0x085fe38b in RecordVersion::fetchVersion (this=0xae326fe8, 
    trans=0xb72843d0) at RecordVersion.cpp:197
#14 0x0856cbde in StorageDatabase::nextRow (this=0xb70c0158, 
    storageTable=0xb74693c8, recordNumber=0, lockForUpdate=false)
    at StorageDatabase.cpp:295
#15 0x08572f6a in StorageTable::next (this=0xb74693c8, recordNumber=0, 
    lockForUpdate=false) at StorageTable.cpp:161
#16 0x0855d2b8 in StorageInterface::rnd_next (this=0xbd69898, buf=0xbd69a70 "")
    at ha_falcon.cpp:653
#17 0x083ff03b in rr_sequential (info=0xbd798dc) at records.cc:390
#18 0x083448a7 in join_init_read_record (tab=0xbd79898) at sql_select.cc:17086
#19 0x08347cf5 in sub_select (join=0xbd7b660, join_tab=0xbd79898, 
    end_of_records=false) at sql_select.cc:16280
#20 0x08354335 in do_select (join=0xbd7b660, fields=0xbd609c4, table=0x0, 
    procedure=0x0) at sql_select.cc:15844
#21 0x0836eee4 in JOIN::exec (this=0xbd7b660) at sql_select.cc:2886
#22 0x08369a8d in mysql_select (thd=0xbd5f5e0, rref_pointer_array=0xbd60a34, 
    tables=0xbd783d8, wild_num=1, fields=@0xbd609c4, conds=0xbd789a8, 
    og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, 
    select_options=2147764736, result=0xbd78af8, unit=0xbd60694, 
    select_lex=0xbd60930) at sql_select.cc:3067
#23 0x0836f1f4 in handle_select (thd=0xbd5f5e0, lex=0xbd60638, 
    result=0xbd78af8, setup_tables_done_option=0) at sql_select.cc:310
#24 0x082d23af in execute_sqlcom_select (thd=0xbd5f5e0, all_tables=0xbd783d8)
    at sql_parse.cc:4949
#25 0x082d3482 in mysql_execute_command (thd=0xbd5f5e0) at sql_parse.cc:2157
#26 0x082dbfbb in mysql_parse (thd=0xbd5f5e0, 
    inBuf=0xbd78248 "SELECT * FROM `A` WHERE `date_key` < 'mfmfxnxnokokibibfdfd'", length=59, found_semicolon=0xa7080e80) at sql_parse.cc:5964
#27 0x082dd13d in dispatch_command (command=COM_QUERY, thd=0xbd5f5e0, 
    packet=0xbd6c389 "", packet_length=59) at sql_parse.cc:1049
#28 0x082de3ce in do_command (thd=0xbd5f5e0) at sql_parse.cc:731
#29 0x082cad73 in handle_one_connection (arg=0xbd5f5e0) at sql_connect.cc:1146
#30 0x0089a45b in start_thread () from /lib/libpthread.so.0
#31 0x007f1c4e in clone () from /lib/libc.so.6