Bug #38039 Assertion lockState == 0 failed in StorageTableShare::deleteTable SyncObject
Submitted: 11 Jul 2008 9:11 Modified: 4 Oct 2008 13:36
Reporter: Philip Stoev Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S1 (Critical)
Version:6.0-falcon OS:Any
Assigned to: Christopher Powers CPU Architecture:Any
Triage: D1 (Critical)

[11 Jul 2008 9:11] Philip Stoev
Description:
When executing a non-concurrent query workload consisting of INSERT/UPDATE/DELETE/SELECT and ALTER ADD/DROP PRIMARY/KEY/UNIQUE, mysqld crashed after just a few queries as follows:

#0  0x00110416 in __kernel_vsyscall ()
#1  0x00581c78 in pthread_kill () from /lib/libpthread.so.0
#2  0x08661e94 in my_write_core (sig=6) at stacktrace.c:307
#3  0x08251c75 in handle_segfault (sig=6) at mysqld.cc:2638
#4  <signal handler called>
#5  0x00110416 in __kernel_vsyscall ()
#6  0x00584fe0 in raise () from /lib/libpthread.so.0
#7  0x084c380c in Error::debugBreak () at Error.cpp:94
#8  0x084c3883 in Error::error (
    string=0x87817b8 "assertion (%s) failed at line %d in file %s\n")
    at Error.cpp:71
#9  0x084c3917 in Error::assertionFailed (text=0x8777d35 "lockState == 0",
    fileName=0x8777d20 "SyncObject.cpp", line=131) at Error.cpp:78
#10 0x084713ba in ~SyncObject (this=0xa7f36120) at SyncObject.cpp:131
#11 0x0846d1b7 in ~StorageTableShare (this=0xa7f36218)
    at StorageTableShare.cpp:90
#12 0x0846bbf2 in StorageTableShare::deleteTable (this=0xa7f36218,
    storageConnection=0xb705c938) at StorageTableShare.cpp:186
#13 0x08469a56 in StorageTable::deleteTable (this=0xb73bc688)
    at StorageTable.cpp:89
#14 0x0845d88c in StorageInterface::delete_table (this=0xa4e7a48,
    tableName=0xa895a8b0 "./test/#sql2-44f-3") at ha_falcon.cpp:980
#15 0x08360dc3 in handler::ha_delete_table (this=0xa4e7a48,
---Type <return> to continue, or q <return> to quit---
    name=0xa895a8b0 "./test/#sql2-44f-3") at handler.cc:3200
#16 0x08366b7e in ha_delete_table (thd=0xa4cef48, table_type=0x9f3a998,
    path=0xa895a8b0 "./test/#sql2-44f-3", db=0xa4d9290 "test",
    alias=0xa895b3a4 "#sql2-44f-3", generate_warning=false) at handler.cc:1906
#17 0x08375c76 in quick_rm_table (base=0x9f3a998, db=0xa4d9290 "test",
    table_name=0xa895b3a4 "#sql2-44f-3", flags=3) at sql_table.cc:1895
#18 0x08377435 in mysql_alter_table (thd=0xa4cef48, new_db=0xa4d9290 "test",
    new_name=0xa4d9068 "A", create_info=0xa895b68c, table_list=0xa4d9090,
    alter_info=0xa895bb74, order_num=0, order=0x0, ignore=false)
    at sql_table.cc:7057
#19 0x0826260d in mysql_execute_command (thd=0xa4cef48) at sql_parse.cc:2631
#20 0x08268f54 in mysql_parse (thd=0xa4cef48,
    inBuf=0xa4d8da8 "ALTER TABLE A DROP KEY int_nokey", length=32,
    found_semicolon=0xa895c260) at sql_parse.cc:5811
#21 0x08269890 in dispatch_command (command=COM_QUERY, thd=0xa4cef48,
    packet=0xa4d0d79 "ALTER TABLE A DROP KEY int_nokey", packet_length=32)
    at sql_parse.cc:1051
#22 0x0826a981 in do_command (thd=0xa4cef48) at sql_parse.cc:724
#23 0x0825a303 in handle_one_connection (arg=0xa4cef48) at sql_connect.cc:1153
#24 0x0057d32f in start_thread () from /lib/libpthread.so.0
#25 0x0049a27e in clone () from /lib/libc.so.6

How to repeat:
A test case will be provided shortly.
[11 Jul 2008 9:30] Philip Stoev
Test case:

--disable_abort_on_error
--source include/have_falcon.inc

DROP TABLE IF EXISTS `A`;
CREATE TABLE `A` (
  `pk` int(11) NOT NULL AUTO_INCREMENT,
  `int_nokey` int(11) NOT NULL,
  `int_key` int(11) NOT NULL,
  `date_key` date NOT NULL,
  `date_nokey` date NOT NULL,
  `time_key` time NOT NULL,
  `time_nokey` time NOT NULL,
  `datetime_key` datetime NOT NULL,
  `datetime_nokey` datetime NOT NULL,
  `varchar_key` varchar(1) DEFAULT NULL,
  `varchar_nokey` varchar(1) DEFAULT NULL,
  PRIMARY KEY (`pk`),
  KEY `int_key` (`int_key`),
  KEY `date_key` (`date_key`),
  KEY `time_key` (`time_key`),
  KEY `datetime_key` (`datetime_key`),
  KEY `varchar_key` (`varchar_key`)
) ENGINE=Falcon;

ALTER TABLE A ADD UNIQUE ( int_nokey );
ALTER TABLE A DROP KEY int_key;
ALTER TABLE A DROP KEY int_nokey;

Please note that the second alter produces an error, and the final alter crashes.
[11 Jul 2008 16:19] Kevin Lewis
Chris, This is a P1 related to your recent push for Add Drop Index, I think.
[16 Jul 2008 10:04] John Embretsen
Before the online add/drop index feature was disabled I got basically the same stacktrace when running this minimal test case:

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a_int INT) Engine = Falcon;
ALTER TABLE t1 ADD KEY (a_int);
ALTER TABLE t1 ADD KEY (a_int);
ALTER TABLE t1 DROP KEY a_int;

==> ERROR 1296 (HY000): Got error 124 'can't find table "TEST.T1"' from Falcon

DROP TABLE t1;

==> ERROR 2013 (HY000): Lost connection to MySQL server during query

This does not fail with the current mysql-6.0-falcon branch (online add/drop disabled). I am planning to add this as a falcon regression test for this bug.
[16 Jul 2008 12:41] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/49820

2752 John H. Embretsen	2008-07-16
      Regression test for issue detected in first implementation of 
      online add/drop index for Falcon, Bug#38039 (Assertion 
      lockState == 0 failed in StorageTableShare::deleteTable SyncObject).
      
      Testing implicit online/offline behavior, i.e. no ONLINE or OFFLINE
      keyword is used.
      
      If online add/drop index is disabled, offline will be used instead. 
      Hence, the test passes in the current falcon tree, and will pass
      if online add/drop index is re-enabled with an appropriate bugfix.
      
      The test will fail if the bug is not fixed when the feature is
      re-enabled (ONLINE is implicit when enabled in the Falcon code).
[16 Jul 2008 12:53] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/49821

2752 John H. Embretsen	2008-07-16
      Regression test for issue detected in first implementation of 
      online add/drop index for Falcon, Bug#38039 (Assertion 
      lockState == 0 failed in StorageTableShare::deleteTable SyncObject).
      
      Testing implicit online/offline behavior, i.e. no ONLINE or OFFLINE
      keyword is used.
      
      If online add/drop index is disabled, offline will be used instead. 
      Hence, the test passes in the current falcon tree, and will pass
      if online add/drop index is re-enabled with an appropriate bugfix.
      
      The test will fail if the bug is not fixed when the feature is
      re-enabled (ONLINE is implicit when enabled in the Falcon code).
[16 Jul 2008 12:58] John Embretsen
Regression test (reviewed by bug reporter) pushed to mysql-6.0-falcon branch.
Setting status back to verified since there is no bugfix yet.
[25 Jul 2008 4:51] Christopher Powers
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/50471

2745 Christopher Powers	2008-07-24
   Bug#38039 Assertion lockState == 0 failed in StorageTableShare::deleteTable
   Bug#38041 Bizarre errors when ALTER ADD/DROP KEY on Falcon tables
   Bug#38043 Deadlock between server and falcon when executing concurrent ALTER
      
   Improvements to online add/drop index:
      
   - StorageTableShare::indexes[] is now a resizeable DenseArray
   - Added sync lock protection to get/set index
   - Call add/drop index from StorageTable rather than StorageTableShare
   - Rename internal Falcon indexes to preserve index name/id mapping after drop
   - Immediately update StorageTableShare::indexes[] after add/drop rather than
     wait for subsequent open()
[25 Jul 2008 4:53] Christopher Powers
Test case passes.
[18 Aug 2008 5:48] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51820

2787 Christopher Powers	2008-08-18
      Bug#38044 Falcon crash in StorageTable::compareKey at StorageTable.cpp:412
      Bug#38043 Deadlock between server and falcon when executing concurrent ALTER + DML
      Bug#38039 Assertion lockState == 0 failed in StorageTableShare::deleteTable SyncObject
      Bug#38041 Bizarre errors when ALTER ADD/DROP KEY on Falcon tables
      
      Improved index synchronization in the StorageInterface:
      - Falcon internal indexes no longer dependent upon server index id
      - Client connections lock StorageTableShare::syncIndexes to protect indexes across calls into StorageInterface
      - Changes to StorageTableShare::indexes are now protected by exclusive lock
[18 Aug 2008 16:55] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51859

2787 Christopher Powers	2008-08-18
      Bug#38044 Falcon crash in StorageTable::compareKey at StorageTable.cpp:412
      Bug#38043 Deadlock between server and falcon when executing concurrent ALTER + DML
      Bug#38039 Assertion lockState == 0 failed in StorageTableShare::deleteTable SyncObject
      Bug#38041 Bizarre errors when ALTER ADD/DROP KEY on Falcon tables
      
      Improved index synchronization in the StorageInterface:
      - Falcon internal indexes no longer dependent upon server index id
      - Client connections lock StorageTableShare::syncIndexes to protect indexes across calls into StorageInterface
      - Changes to StorageTableShare::indexes are now protected by exclusive lock
[18 Aug 2008 17:06] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/51863

2787 Christopher Powers	2008-08-18
      Bug#38044 Falcon crash in StorageTable::compareKey at StorageTable.cpp:412
      Bug#38043 Deadlock between server and falcon when executing concurrent ALTER + DML
      Bug#38039 Assertion lockState == 0 failed in StorageTableShare::deleteTable SyncObject
      Bug#38041 Bizarre errors when ALTER ADD/DROP KEY on Falcon tables
      
      Improved index synchronization in the StorageInterface:
      - Falcon internal indexes no longer dependent upon server index id
      - Client connections lock StorageTableShare::syncIndexes to protect indexes across calls into StorageInterface
      - Changes to StorageTableShare::indexes are now protected by exclusive lock
[28 Aug 2008 4:36] Bugs System
Pushed into 6.0.7-alpha  (revid:cpowers@mysql.com-20080818054529-s9swke0xd15rx1fu) (version source revid:vvaintroub@mysql.com-20080827094944-sh582y3m7duhzrpj) (pib:3)
[13 Sep 2008 22:49] Bugs System
Pushed into 6.0.7-alpha  (revid:john.embretsen@sun.com-20080716124047-qw0hob61zsh4y941) (version source revid:davi.arnaut@sun.com-20080812141852-8e6knbqclpfd8irn) (pib:3)
[4 Oct 2008 13:36] Jon Stephens
Documented bugfix as follows, in the 6.0.7 changelog:

        Dropping a Falcon table, followed by re-creating the table and then
        adding indexes to the re-created table, could cause spurious errors or
        possibly a crash of the server.