Bug #38567 Falcon crash in Record::getEncodedRecord at Record.cpp:743
Submitted: 5 Aug 2008 11:26 Modified: 13 Dec 2008 9:04
Reporter: Philip Stoev Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:6.0-falcon-team OS:Any
Assigned to: Christopher Powers CPU Architecture:Any
Tags: F_CHILL THAW
Triage: Triaged: D1 (Critical)

[5 Aug 2008 11:26] Philip Stoev
Description:
When executing an insert/update workload under falcon_index_chill_threshold=4K  and falcon_record_chill_threshold=4K, Falcon crashed as follows:

#0  0x00110416 in __kernel_vsyscall ()
#1  0x00581c78 in pthread_kill () from /lib/libpthread.so.0
#2  0x087bdb43 in my_write_core (sig=11) at stacktrace.c:307
#3  0x082acd2b in handle_segfault (sig=11) at mysqld.cc:2657
#4  <signal handler called>
#5  0x085b220e in Record::getEncodedRecord (this=0xa69dd048) at Record.cpp:743
#6  0x085218f5 in StorageTable::setRecord (this=0xb7447dd8, newRecord=0xa69dd048, locked=true) at StorageTable.cpp:336
#7  0x0851b99e in StorageDatabase::nextIndexed (this=0xb708b130, storageTable=0xb7447dd8, recordBitmap=0xb71d4cb8, recordNumber=142, lockForUpdate=true)
    at StorageDatabase.cpp:440
#8  0x08521e67 in StorageTable::nextIndexed (this=0xb7447dd8, recordNumber=142, lockForUpdate=true) at StorageTable.cpp:182
#9  0x08514fcc in StorageInterface::index_next (this=0x9b2e0d8, buf=0x9b2e2a0 "+") at ha_falcon.cpp:1597
#10 0x083d028a in handler::read_range_next (this=0x9b2e0d8) at handler.cc:4938
#11 0x083ce524 in handler::multi_range_read_next (this=0x9b2e0d8, range_info=0xa72af170) at handler.cc:4232
#12 0x083aeeef in QUICK_RANGE_SELECT::get_next (this=0x9cb76e8) at opt_range.cc:8518
#13 0x083c8897 in rr_quick (info=0xa72af33c) at records.cc:298
#14 0x0835f2d4 in mysql_update (thd=0x9d7abf8, table_list=0x9e2ba60, fields=@0x9d7c090, values=@0x9d7c2c0, conds=0x9e2c0f8, order_num=0, order=0x0,
    limit=18446744073709551522, handle_duplicates=DUP_ERROR, ignore=false) at sql_update.cc:469
#15 0x082bf817 in mysql_execute_command (thd=0x9d7abf8) at sql_parse.cc:2846
#16 0x082c63e1 in mysql_parse (thd=0x9d7abf8, inBuf=0x9e2b740 "UPDATE B AS X SET `int_key` = 206 WHERE X . `int_key` < 147", length=59,
    found_semicolon=0xa72b0270) at sql_parse.cc:5809
#17 0x082c6e55 in dispatch_command (command=COM_QUERY, thd=0x9d7abf8, packet=0x9d50d31 "UPDATE B AS X SET `int_key` = 206 WHERE X . `int_key` < 147",
    packet_length=59) at sql_parse.cc:1050
#18 0x082c8145 in do_command (thd=0x9d7abf8) at sql_parse.cc:723
#19 0x082b5e23 in handle_one_connection (arg=0x9d7abf8) at sql_connect.cc:1153
#20 0x0057d32f in start_thread () from /lib/libpthread.so.0
#21 0x0049a27e in clone () from /lib/libc.so.6

The crash is here:

738             if (state == recChilled)
739                     thaw();
740
741             ASSERT(encoding == shortVector);
742
743             return data.record + ((USHORT*) data.record)[0]; <<<<<< HERE
744     }
745
746     const UCHAR* Record::getEncoding(int index)
747     {

(gdb) print data
$1 = {record = 0x0}

How to repeat:
If this is repeatable, a test case will be provided.
[5 Aug 2008 12:40] Philip Stoev
To reproduce this bug, please clone the mysql-test-extra-6.0 repository and execute

$ cd mysql-test-extra-6.0/mysqltest/gentest
$ perl runall.pl \
  --basedir=/path/to/6.0-falcon/ \
  --engine=falcon \
  --grammar=conf/chill_thaw.yy \
  --threads=20 --queries=100000 \
  --mysqld=--falcon_debug_mask=7167 \
  --falcon_record_chill_threshold=32K

This crash (or another chill/thaw issue) should happen within less than a mintute, after a few dozen chill/thaw operations.
[19 Aug 2008 20:57] Kevin Lewis
Related to falcon_record_chill_threshold=32K
[24 Oct 2008 5:09] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/56961

2880 Christopher Powers	2008-10-24
      Bug#36631,"Assertion in SerialLogControl::nextRecord"
      Bug#38541,"Falcon RecordVersion::thaw assertion """"bytesRestored > 0 || data.record == NULL"""
      Bug#38567,"Falcon crash in  Record::getEncodedRecord at Record.cpp:743"
      Bug#38569,"Falcon assertion in SRLUpdateIndex::thaw : type == srlUpdateIndex"
      Bug#39694,"Crash in StorageTable::setRecord during falcon_chill_thaw test"
      Bug#39695,"Crash in SRLUpdateRecords::thaw during falcon_chill_thaw test"
      Bug#39696,"Assertion in Table.cpp (dup->state == recDeleted) fails during falcon_chill_thaw"
      
      Fix 1 of 3 for chill/thaw bugs.
      
      Modified Record::hasRecord() to thaw if necessary, removed redundant record thaws, added exception handling
[24 Oct 2008 5:38] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/56962

2881 Christopher Powers	2008-10-24
      Bug#36631,"Assertion in SerialLogControl::nextRecord"
      Bug#38541,"Falcon RecordVersion::thaw assertion """"bytesRestored > 0 || data.record == NULL"""
      Bug#38567,"Falcon crash in  Record::getEncodedRecord at Record.cpp:743"
      Bug#38569,"Falcon assertion in SRLUpdateIndex::thaw : type == srlUpdateIndex"
      Bug#39694,"Crash in StorageTable::setRecord during falcon_chill_thaw test"
      Bug#39695,"Crash in SRLUpdateRecords::thaw during falcon_chill_thaw test"
      Bug#39696,"Assertion in Table.cpp (dup->state == recDeleted) fails during falcon_chill_thaw"
      
      Fix 2 of 3 for chill/thaw bugs.
      
      Synchronized record thaw operations using a pool of sync objects in the associated table.
      Record chill operations do not need to be synchronized because chills are performed in one place, and only by the active Transaction.
[26 Oct 2008 15:54] Kevin Lewis
Two patches approved to problems with concurent thaws.
1) enhance hasRecord to thaw when needed.
2) serialize thaws with a hash table of SyncOjects.
[28 Oct 2008 8:10] Bugs System
Pushed into 6.0.8-alpha  (revid:cpowers@mysql.com-20081024053538-6y1ma8lydl93oxza) (version source revid:alik@mysql.com-20081028074021-o49ijs3x4pxj1gbm) (pib:5)
[13 Dec 2008 9:04] MC Brown
No documentation needed (internal change only)