Bug #40690 Falcon: Deferred index thaw crashes in addNode()
Submitted: 13 Nov 2008 9:45 Modified: 13 Dec 2008 10:16
Reporter: Christopher Powers
Status: Closed
Category:Server: Falcon Severity:S1 (Critical)
Version:6.0 OS:Any
Assigned to: Christopher Powers Target Version:6.0-beta
Tags: test failure, pb2, F_CHILL THAW
Triage: Needs Triage: D1 (Critical)

[13 Nov 2008 9:45] Christopher Powers
Description:
Deferred index thaw operations sometime crash during the falcon_chill_thaw stress test.
The crash is typically in either DeferredIndex::addNode() or DeferredIndex::compare().

This stack trace shows an index scan that crashed due to a segfault in
DeferredIndex::compare():

#4  <signal handler called>
#5  0x0859436a in DeferredIndex::compare (this=0xb727e398, node1=0xb727e590,
node2=0x9c876bf4) at DeferredIndex.cpp:458
#6  0x08594ed5 in DeferredIndex::addNode (this=0xb727e398, indexKey=0xa91ad538,
recordNumber=0) at DeferredIndex.cpp:157
#7  0x085ec9ea in SRLUpdateIndex::thaw (this=0xa91b0430, deferredIndex=0xb727e398) at
SRLUpdateIndex.cpp:329
#8  0x08563889 in Transaction::thaw (this=0xb71e84e8, deferredIndex=0xb727e398)at
Transaction.cpp:605
#9  0x085941f7 in DeferredIndex::scanIndex (this=0xb727e398, lowKey=0x0,
highKey=0xb61a3ac4, searchFlags=4, bitmap=0xb7216420) at DeferredIndex.cpp:794
#10 0x085a811d in Index::scanIndex (this=0xb711aa78, lowKey=0x0, highKey=0xb61a3ac4,
searchFlags=4, transaction=0xb71e84e8, bitmap=0xb7216420) at Index.cpp:447
#11 0x0853ef65 in StorageDatabase::indexScan (this=0xb70ab130, index=0xb711aa78,
lower=0x0, upper=0xb61a3ac0, searchFlags=4, storageConnection=0xb728ca40,
bitmap=0xb7216420) at StorageDatabase.cpp:796
#12 0x08545d3e in StorageTable::indexScan (this=0xb61a1048, indexOrder=0) at
StorageTable.cpp:243
#13 0x0852fdf9 in StorageInterface::read_range_first (this=0xb9b11b0, start_key=0x0,
end_key=0xb9b1254, eq_range_arg=false, sorted=false) at ha_falcon.cpp:1618
#14 0x083e2551 in handler::multi_range_read_next (this=0xb9b11b0, range_info=0xa91b08b0)
at handler.cc:4255
#15 0x083c41cd in QUICK_RANGE_SELECT::get_next (this=0xb9d8f08) at opt_range.cc:8596
#16 0x083dbd75 in rr_quick (info=0xb9f764c) at records.cc:322
#17 0x0832c2dd in join_init_read_record (tab=0xb9f7608) at sql_select.cc:14545
#18 0x0832e4ad in sub_select (join=0xb9ce1a0, join_tab=0xb9f7608, end_of_records=false)
at sql_select.cc:13711
#19 0x08335937 in do_select (join=0xb9ce1a0, fields=0xb9b3f58, table=0x0, procedure=0x0)
at sql_select.cc:13458

How to repeat:
Branch or checkout the mysql-test-extra-6.0 bazaar tree and run:

$ cd mysql-test\gentest
$ perl runall.pl \
 --basedir=path\to\mysql-6.0-falcon-team \
 --reporters=Deadlock,ErrorLog,Backtrace \
 --mysqld=--loose-falcon-lock-wait-timeout=1 \
 --engine=Falcon \
 --grammar=conf/falcon_chill_thaw.yy \ 
 --queries=100000 \
 --mysqld=--falcon-index-chill-threshold=4K \ 
 --duration=1200

Suggested fix:
The segfault was caused by a stale index node, DeferredIndex::maxValue. Several fields in
DeferredIndex become invalid when the index is chilled, and must therefore be reset. The
min/max fields must also be reset when the index is chilled:

DeferredIndex::chill(Dbb* dbb)
{
   ...
   minValue = NULL;
   maxValue = NULL;
   haveMinValue = true;
   haveMaxValue = true;
}
[13 Nov 2008 10:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/58610

2909 Christopher Powers	2008-11-13
      Bug#40690, "Falcon: Deferred index thaw crashes in addNode()"
      Bug#40691, "Falcon: Chill/thaw operations do not lock SerialLog::syncWrite"
[14 Nov 2008 16:42] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/58805

2913 Christopher Powers	2008-11-14
      Bug#40690, "Falcon: Deferred index thaw crashes in addNode()"
      Bug#40691, "Falcon: Chill/thaw operations do not lock SerialLog::syncWrite"
      
      Closed synchronization gaps for record+index chill/thaw
[20 Nov 2008 13:56] Bugs System
Pushed into 6.0.9-alpha  (revid:cpowers@mysql.com-20081114153844-wgyg2mm8ihig75el)
(version source revid:cpowers@mysql.com-20081114153844-wgyg2mm8ihig75el) (pib:5)
[13 Dec 2008 10:16] MC Brown
No documentation needed (internal change only)