Bug #40690 Falcon: Deferred index thaw crashes in addNode()
Submitted: 13 Nov 2008 8:45 Modified: 13 Dec 2008 9:16
Reporter: Christopher Powers Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:6.0 OS:Any
Assigned to: Christopher Powers CPU Architecture:Any
Tags: F_CHILL THAW, pb2, test failure

[13 Nov 2008 8:45] Christopher Powers
Description:
Deferred index thaw operations sometime crash during the falcon_chill_thaw stress test. The crash is typically in either DeferredIndex::addNode() or DeferredIndex::compare().

This stack trace shows an index scan that crashed due to a segfault in DeferredIndex::compare():

#4  <signal handler called>
#5  0x0859436a in DeferredIndex::compare (this=0xb727e398, node1=0xb727e590, node2=0x9c876bf4) at DeferredIndex.cpp:458
#6  0x08594ed5 in DeferredIndex::addNode (this=0xb727e398, indexKey=0xa91ad538, recordNumber=0) at DeferredIndex.cpp:157
#7  0x085ec9ea in SRLUpdateIndex::thaw (this=0xa91b0430, deferredIndex=0xb727e398) at SRLUpdateIndex.cpp:329
#8  0x08563889 in Transaction::thaw (this=0xb71e84e8, deferredIndex=0xb727e398)at Transaction.cpp:605
#9  0x085941f7 in DeferredIndex::scanIndex (this=0xb727e398, lowKey=0x0, highKey=0xb61a3ac4, searchFlags=4, bitmap=0xb7216420) at DeferredIndex.cpp:794
#10 0x085a811d in Index::scanIndex (this=0xb711aa78, lowKey=0x0, highKey=0xb61a3ac4, searchFlags=4, transaction=0xb71e84e8, bitmap=0xb7216420) at Index.cpp:447
#11 0x0853ef65 in StorageDatabase::indexScan (this=0xb70ab130, index=0xb711aa78, lower=0x0, upper=0xb61a3ac0, searchFlags=4, storageConnection=0xb728ca40, bitmap=0xb7216420) at StorageDatabase.cpp:796
#12 0x08545d3e in StorageTable::indexScan (this=0xb61a1048, indexOrder=0) at StorageTable.cpp:243
#13 0x0852fdf9 in StorageInterface::read_range_first (this=0xb9b11b0, start_key=0x0, end_key=0xb9b1254, eq_range_arg=false, sorted=false) at ha_falcon.cpp:1618
#14 0x083e2551 in handler::multi_range_read_next (this=0xb9b11b0, range_info=0xa91b08b0) at handler.cc:4255
#15 0x083c41cd in QUICK_RANGE_SELECT::get_next (this=0xb9d8f08) at opt_range.cc:8596
#16 0x083dbd75 in rr_quick (info=0xb9f764c) at records.cc:322
#17 0x0832c2dd in join_init_read_record (tab=0xb9f7608) at sql_select.cc:14545
#18 0x0832e4ad in sub_select (join=0xb9ce1a0, join_tab=0xb9f7608, end_of_records=false) at sql_select.cc:13711
#19 0x08335937 in do_select (join=0xb9ce1a0, fields=0xb9b3f58, table=0x0, procedure=0x0) at sql_select.cc:13458

How to repeat:
Branch or checkout the mysql-test-extra-6.0 bazaar tree and run:

$ cd mysql-test\gentest
$ perl runall.pl \
 --basedir=path\to\mysql-6.0-falcon-team \
 --reporters=Deadlock,ErrorLog,Backtrace \
 --mysqld=--loose-falcon-lock-wait-timeout=1 \
 --engine=Falcon \
 --grammar=conf/falcon_chill_thaw.yy \ 
 --queries=100000 \
 --mysqld=--falcon-index-chill-threshold=4K \ 
 --duration=1200

Suggested fix:
The segfault was caused by a stale index node, DeferredIndex::maxValue. Several fields in DeferredIndex become invalid when the index is chilled, and must therefore be reset. The min/max fields must also be reset when the index is chilled:

DeferredIndex::chill(Dbb* dbb)
{
   ...
   minValue = NULL;
   maxValue = NULL;
   haveMinValue = true;
   haveMaxValue = true;
}
[13 Nov 2008 9:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/58610

2909 Christopher Powers	2008-11-13
      Bug#40690, "Falcon: Deferred index thaw crashes in addNode()"
      Bug#40691, "Falcon: Chill/thaw operations do not lock SerialLog::syncWrite"
[14 Nov 2008 15:42] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/58805

2913 Christopher Powers	2008-11-14
      Bug#40690, "Falcon: Deferred index thaw crashes in addNode()"
      Bug#40691, "Falcon: Chill/thaw operations do not lock SerialLog::syncWrite"
      
      Closed synchronization gaps for record+index chill/thaw
[20 Nov 2008 12:56] Bugs System
Pushed into 6.0.9-alpha  (revid:cpowers@mysql.com-20081114153844-wgyg2mm8ihig75el) (version source revid:cpowers@mysql.com-20081114153844-wgyg2mm8ihig75el) (pib:5)
[13 Dec 2008 9:16] MC Brown
No documentation needed (internal change only)