Bug #38044 Falcon crash in StorageTable::compareKey at StorageTable.cpp:412
Submitted: 11 Jul 2008 11:01 Modified: 4 Oct 2008 13:40
Reporter: Philip Stoev Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:6.0-falcon OS:Any
Assigned to: Christopher Powers CPU Architecture:Any
Triage: D1 (Critical)

[11 Jul 2008 11:01] Philip Stoev
Description:
When executing a concurrent workload involving ALTER ADD/DROP key and DML statements, mysqld crashed as follows:

#0  0x00110416 in __kernel_vsyscall ()
#1  0x00581c78 in pthread_kill () from /lib/libpthread.so.0
#2  0x08661f04 in my_write_core (sig=11) at stacktrace.c:307
#3  0x08251c75 in handle_segfault (sig=11) at mysqld.cc:2638
#4  <signal handler called>
#5  0x08468c2e in StorageTable::compareKey (this=0xb74c9248, key=0x9255518 "", keyLength=4) at StorageTable.cpp:412
#6  0x0845c371 in StorageInterface::index_next_same (this=0xa7c0b6d0, buf=0xa7c0b888 "ะช", key=0x9255518 "", key_len=4) at ha_falcon.cpp:1659
#7  0x082bef02 in join_read_next_same (info=0x92553b8) at sql_select.cc:14413
#8  0x082c2177 in sub_select (join=0x91ee5b8, join_tab=0x9255374, end_of_records=false) at sql_select.cc:13646
#9  0x082c1f3f in evaluate_join_record (join=0x91ee5b8, join_tab=0x92551d0, error=0) at sql_select.cc:13864
#10 0x082c218e in sub_select (join=0x91ee5b8, join_tab=0x92551d0, end_of_records=false) at sql_select.cc:13647
#11 0x082ce441 in do_select (join=0x91ee5b8, fields=0x924c1a4, table=0x0, procedure=0x0) at sql_select.cc:13387
#12 0x082dfc6f in JOIN::exec (this=0x91ee5b8) at sql_select.cc:2811
#13 0x082db671 in mysql_select (thd=0x924ad30, rref_pointer_array=0x924c214, tables=0x9253a10, wild_num=0, fields=@0x924c1a4, conds=0x92548a8, og_num=0,
    order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2148289024, result=0x92549f8, unit=0x924be80, select_lex=0x924c110)
    at sql_select.cc:3001
#14 0x082dff27 in handle_select (thd=0x924ad30, lex=0x924be24, result=0x92549f8, setup_tables_done_option=0) at sql_select.cc:302
#15 0x0825fa1f in execute_sqlcom_select (thd=0x924ad30, all_tables=0x9253a10) at sql_parse.cc:4847
#16 0x082610c6 in mysql_execute_command (thd=0x924ad30) at sql_parse.cc:2015
#17 0x08268f54 in mysql_parse (thd=0x924ad30,
    inBuf=0x9253810 "SELECT X . int_key FROM E AS X LEFT JOIN D AS Y ON ( X . int_key = Y . int_nokey ) WHERE X . int_nokey < 30", length=107,
    found_semicolon=0xa7bce260) at sql_parse.cc:5811
#18 0x08269890 in dispatch_command (command=COM_QUERY, thd=0x924ad30, packet=0xa7c05fb9 "\0010\002", packet_length=108) at sql_parse.cc:1051
#19 0x0826a981 in do_command (thd=0x924ad30) at sql_parse.cc:724
#20 0x0825a303 in handle_one_connection (arg=0x924ad30) at sql_connect.cc:1153
#21 0x0057d32f in start_thread () from /lib/libpthread.so.0
#22 0x0049a27e in clone () from /lib/libc.so.6

The crash is here:

407                     for (const UCHAR *p = key, *end = key + keyLength; p < end && segmentNumber < indexDesc->numberSegments; ++segmentNumber)
408                             {
409                             StorageSegment *segment = indexDesc->segments + segmentNumber;
410                             int nullFlag = (segment->nullBit) ? *p++ : 0;
411                             Value keyValue;
412                             int len = storageDatabase->getSegmentValue(segment, p, &keyValue, index->fields[segmentNumber]); <<<<<<<<<<<HERE
413                             Field *field = index->fields[segmentNumber];
414
415                             if (nullFlag)
416                                     {

(gdb) print index->fields
$4 = (class Nfs::Field **) 0xeeeeeeee

How to repeat:
If this happens again, a repeatable test case will be provided.
[11 Jul 2008 17:37] Kevin Lewis
Chris,  This call stack and debug output that Philip so competently provided shows that the index pointer is now pointing to invalid memory.  So the index has been dropped while another client thread still has a pointer to it.
[25 Jul 2008 4:56] Christopher Powers
This is reproducible using the short test case from Bug#38043:

perl runall.pl --basedir=/path/to/mysql-6.0-falcon --engine=falcon \
  --grammar=path/to/bug38043.yy --mysqld=--log-output=file --threads=2
[16 Aug 2008 22:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51810

2788 Christopher Powers	2008-08-16
      Bug#38044 Falcon crash in StorageTable::compareKey
      
      Removed debugging code
      Added StorageTableShare::syncIndexes
[18 Aug 2008 3:56] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51817

2790 Christopher Powers	2008-08-17
      Bug#38044 Falcon crash in StorageTable::compareKey
      - Removed debug code
      - Fixed compiler warnings
      - Fixed bug in StorageTableShare::getIndex() that caused optimization to fail
[18 Aug 2008 4:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51818

2790 Christopher Powers	2008-08-17
      Bug#38044 Falcon crash in StorageTable::compareKey
      - Removed debug code
      - Fixed compiler warnings
      - Fixed bug in StorageTableShare::getIndex() that caused optimization to fail
[18 Aug 2008 4:47] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51819

2790 Christopher Powers	2008-08-17
      Bug#38044 Falcon crash in StorageTable::compareKey
      - Removed debug code
      - Fixed compiler warnings
      - Fixed bug in StorageTableShare::getIndex() that caused optimization to fail
[18 Aug 2008 5:48] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51820

2787 Christopher Powers	2008-08-18
      Bug#38044 Falcon crash in StorageTable::compareKey at StorageTable.cpp:412
      Bug#38043 Deadlock between server and falcon when executing concurrent ALTER + DML
      Bug#38039 Assertion lockState == 0 failed in StorageTableShare::deleteTable SyncObject
      Bug#38041 Bizarre errors when ALTER ADD/DROP KEY on Falcon tables
      
      Improved index synchronization in the StorageInterface:
      - Falcon internal indexes no longer dependent upon server index id
      - Client connections lock StorageTableShare::syncIndexes to protect indexes across calls into StorageInterface
      - Changes to StorageTableShare::indexes are now protected by exclusive lock
[18 Aug 2008 16:56] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51859

2787 Christopher Powers	2008-08-18
      Bug#38044 Falcon crash in StorageTable::compareKey at StorageTable.cpp:412
      Bug#38043 Deadlock between server and falcon when executing concurrent ALTER + DML
      Bug#38039 Assertion lockState == 0 failed in StorageTableShare::deleteTable SyncObject
      Bug#38041 Bizarre errors when ALTER ADD/DROP KEY on Falcon tables
      
      Improved index synchronization in the StorageInterface:
      - Falcon internal indexes no longer dependent upon server index id
      - Client connections lock StorageTableShare::syncIndexes to protect indexes across calls into StorageInterface
      - Changes to StorageTableShare::indexes are now protected by exclusive lock
[18 Aug 2008 17:06] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51863

2787 Christopher Powers	2008-08-18
      Bug#38044 Falcon crash in StorageTable::compareKey at StorageTable.cpp:412
      Bug#38043 Deadlock between server and falcon when executing concurrent ALTER + DML
      Bug#38039 Assertion lockState == 0 failed in StorageTableShare::deleteTable SyncObject
      Bug#38041 Bizarre errors when ALTER ADD/DROP KEY on Falcon tables
      
      Improved index synchronization in the StorageInterface:
      - Falcon internal indexes no longer dependent upon server index id
      - Client connections lock StorageTableShare::syncIndexes to protect indexes across calls into StorageInterface
      - Changes to StorageTableShare::indexes are now protected by exclusive lock
[28 Aug 2008 4:37] Bugs System
Pushed into 6.0.7-alpha  (revid:cpowers@mysql.com-20080818054529-s9swke0xd15rx1fu) (version source revid:vvaintroub@mysql.com-20080827094944-sh582y3m7duhzrpj) (pib:3)
[14 Sep 2008 4:07] Bugs System
Pushed into 6.0.6-alpha  (revid:cpowers@mysql.com-20080818054529-s9swke0xd15rx1fu) (version source revid:hakan@mysql.com-20080716175219-8unfm2ikarselqpw) (pib:3)
[4 Oct 2008 13:40] Jon Stephens
Documented bugfix in the 6.0.6 changelog as follows:

        Concurrent adding or dropping of indexes and execution of DML statements 
        on a Falcon table could cause the server to crash.