Bug #42862 Crash on failed attempt to open a children of a merge table
Submitted: 14 Feb 2009 20:35 Modified: 7 Mar 2010 12:55
Reporter: Davi Arnaut (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Merge storage engine Severity:S2 (Serious)
Version:6.0-bugteam OS:Any
Assigned to: Konstantin Osipov CPU Architecture:Any

[14 Feb 2009 20:35] Davi Arnaut
Description:
There is a comment in sql_base.cc

 close_tables_for_reopen(thd, start, (action == OT_BACK_OFF_AND_RETRY));
 /*
    Here we rely on the fact that 'tables' still points to the valid
    TABLE_LIST element. Altough currently this assumption is valid
    it may change in future.
 */
  DEBUG_SYNC(thd, "after_close_thread_tables");
  if (recover_from_failed_open_table_attempt(thd, tables, action))

Future has came.. It actually does not point to a valid TABLE_LIST if the failed open was for a children of a merge table and a flush tables sneaks between the close tables and recover.

How to repeat:
Test case:

connect (con1,localhost,root,,);
connect (con2,localhost,root,,);

create table t1 (a int not null) engine=csv;
create table t2 (a int not null) engine=csv;
create table t3 (a int not null) engine=MERGE UNION=(t1,t2);

--connection con1
flush tables;
--remove_file $MYSQLTEST_VARDIR/master-data/test/t2.CSM
set debug_sync='after_close_thread_tables SIGNAL flush WAIT_FOR flushed';
--send select * from t3;

--connection con2
set debug_sync= 'now WAIT_FOR flush';
flush tables;
set debug_sync= 'now SIGNAL flushed';

--conneciton con1
--reap

Sync point:

=== modified file 'sql/sql_base.cc'
--- sql/sql_base.cc    2008-11-06 18:39:27 +0000
+++ sql/sql_base.cc    2008-11-18 14:50:37 +0000
@@ -3744,6 +3744,7 @@ int open_tables(THD *thd, TABLE_LIST **s
           TABLE_LIST element. Altough currently this assumption is valid
           it may change in future.
         */
+        DEBUG_SYNC(thd, "after_close_thread_tables");
         if (recover_from_failed_open_table_attempt(thd, tables, action))
         {
           result= -1;
[15 Feb 2009 8:00] MySQL Verification Team
Davi, do you think this is the same as bug #41212 ?
[15 Feb 2009 9:07] Sveta Smirnova
Thank you for the report.

I can not repeat described behavior:

mysqltest: At line 22: query 'reap' failed: 1168: Unable to open underlying table which is differently defined or of non-MyISAM type or doesn't exist

The result from queries just before the failure was:
create table t1 (a int not null) engine=csv;
create table t2 (a int not null) engine=csv;
create table t3 (a int not null) engine=MERGE UNION=(t1,t2);
flush tables;
set debug_sync='after_close_thread_tables SIGNAL flush WAIT_FOR flushed';
select * from t3;;
set debug_sync= 'now WAIT_FOR flush';
Warnings:
Warning 1728    debug sync point wait timed out
flush tables;
set debug_sync= 'now SIGNAL flushed';

Probably test case should be corrected?
[15 Feb 2009 13:50] Davi Arnaut
The root cause of this crash might be similar to Bug#41212

Sorry for the old test case. This one should yield a crash using latest 6.0-bugteam:

--source include/have_csv.inc

connect (con1,localhost,root,,);
connect (con2,localhost,root,,);

create table t1 (a int not null) engine=csv;
create table t2 (a int not null) engine=csv;
create table t3 (a int not null) engine=MERGE UNION=(t1,t2);

connection con1;
flush tables;
let $MYSQLD_DATADIR= `select @@datadir`;
--remove_file $MYSQLD_DATADIR/test/t2.CSM
set debug_sync='after_close_thread_tables SIGNAL flush WAIT_FOR flushed';
--send select * from t3;

connection con2;
set debug_sync= 'now WAIT_FOR flush';
flush tables;
set debug_sync= 'now SIGNAL flushed';

connection con1;
--reap
[15 Feb 2009 18:40] Valeriy Kravchuk
Same result as Sveta got, with latest 6.0.10 from bzr:

openxs@suse:/home2/openxs/dbs/6.0/mysql-test> ./mysql-test-run.pl bug42862
Logging: ./mysql-test-run.pl  bug42862
090405 15:07:02 [Warning] Forcing shutdown of 2 plugins
MySQL Version 6.0.10
Checking supported features...
 - using ndbcluster when necessary, mysqld supports it
 - SSL connections supported
 - binaries are debug compiled
Collecting tests...
vardir: /home2/openxs/dbs/6.0/mysql-test/var
Checking leftover processes...
Removing old var directory...
Creating var directory '/home2/openxs/dbs/6.0/mysql-test/var'...
Installing system database...
Using server port 1025
worker[1] Using MTR_BUILD_THREAD 250, with reserved ports 12500..12509

==============================================================================

TEST                                      RESULT   TIME (ms)
------------------------------------------------------------

main.bug42862                            [ fail ]
        Test ended at 2009-04-05 15:12:26

CURRENT_TEST: main.bug42862
mysqltest: At line 23: query 'reap' failed: 1168: Unable to open underlying table which is differently defined or of non-MyISAM type or doesn't exist

The result from queries just before the failure was:
create table t1 (a int not null) engine=csv;
create table t2 (a int not null) engine=csv;
create table t3 (a int not null) engine=MERGE UNION=(t1,t2);
flush tables;
set debug_sync='after_close_thread_tables SIGNAL flush WAIT_FOR flushed';
select * from t3;;
set debug_sync= 'now WAIT_FOR flush';
Warnings:
Warning 1728    debug sync point wait timed out
flush tables;
set debug_sync= 'now SIGNAL flushed';

Warnings from just before the error:
Error 1194 Table 't2' is marked as crashed and should be repaired

 - saving '/home2/openxs/dbs/6.0/mysql-test/var/log/main.bug42862/' to '/home2/openxs/dbs/6.0/mysql-test/var/log/main.bug42862/'
------------------------------------------------------------
The servers were restarted 0 times
Spent 0.000 of 326 seconds executing testcases

Failed 1/1 tests, 0.00% were successful.

Failing test(s): main.bug42862

The log files in var/log may give you some hint of what went wrong.

If you want to report this error, please read first the documentation
at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html

mysql-test-run: *** ERROR: there were failing test cases
[16 Feb 2009 10:07] Sveta Smirnova
Same for me: can not repeat crash with new test case either.
[16 Feb 2009 10:46] Sveta Smirnova
Thank you for the feedback.

Crash verified with version mysql-6.0-bugteam. With main tree bug is not repeatable.
[16 Feb 2009 10:47] Sveta Smirnova
Backtrace:

#0  0x002ce402 in __kernel_vsyscall ()
#0  0x002ce402 in __kernel_vsyscall ()
#1  0x0046264f in pthread_kill () from /lib/libpthread.so.0
#2  0x0881050f in my_write_core (sig=11) at stacktrace.c:309
#3  0x082bab54 in handle_segfault (sig=11) at mysqld.cc:2689
#4  <signal handler called>
#5  0x082b599b in mdl_set_lock_type (lock_data=0x8f8f8f8f, lock_type=MDL_EXCLUSIVE) at mdl.h:169
#6  0x08324945 in recover_from_failed_open_table_attempt (thd=0x9a3d600, table=0x9a864c0, action=OT_REPAIR) at sql_base.cc:3515
#7  0x08326a07 in open_tables (thd=0x9a3d600, start=0x9cb75c64, counter=0x9cb75c50, flags=0) at sql_base.cc:3770
#8  0x08327061 in open_and_lock_tables_derived (thd=0x9a3d600, tables=0x9a80b80, derived=true, flags=0) at sql_base.cc:4189
#9  0x082db096 in open_and_lock_tables (thd=0x9a3d600, tables=0x9a80b80) at ../../sql/mysql_priv.h:1570
#10 0x082c9a91 in execute_sqlcom_select (thd=0x9a3d600, all_tables=0x9a80b80) at sql_parse.cc:4722
#11 0x082cf816 in mysql_execute_command (thd=0x9a3d600) at sql_parse.cc:2063
#12 0x082d8669 in mysql_parse (thd=0x9a3d600, inBuf=0x9a80818 "select * from t3", length=16, found_semicolon=0x9cb76f20) at sql_parse.cc:5752
#13 0x082d90ab in dispatch_command (command=COM_QUERY, thd=0x9a3d600, packet=0x9a748f9 "select * from t3;", packet_length=17) at sql_parse.cc:1009
#14 0x082da40f in do_command (thd=0x9a3d600) at sql_parse.cc:691
#15 0x082c82b7 in handle_one_connection (arg=0x9a3d600) at sql_connect.cc:1146
#16 0x0045fbd4 in start_thread () from /lib/libpthread.so.0
#17 0x003b74fe in clone () from /lib/libc.so.6
[16 Feb 2009 11:09] Davi Arnaut
Crashes on 6.0 main for me too, but crashing on my machine and not on others is essentially due to the undefined behavior nature of invalid access to memory. Valgrind run of the test case will probably show up warnings if it does not crash.
[7 Aug 2009 18:55] Konstantin Osipov
See also Bug#46610
[13 Aug 2009 15:16] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/80766

2857 Konstantin Osipov	2009-08-13
      A fix and a test case for Bug#46610 " 	MySQL 5.4.4: MyISAM MRG engine crash 
      on auto-repair of child".
      Also fixes Bug#42862 "Crash on failed attempt to open a children of a 
      merge table".
      
      MERGE engine needs to extend the global table list
      with TABLE_LIST elements for child tables,
      so that they are opened and locked.
      Previously these table list elements were allocated
      in memory of ha_myisammrg object (MERGE engine handler).
      That would lead to access to freed memory in 
      recover_from_failed_open_table_attempt(), which would
      try to recover a MERGE table child (MyISAM table)
      and use for that TABLE_LIST of that child.
      But by the time recover_from_failed_open_table_attempt()
      is invoked, ha_myisammrg object that owns this
      TABLE_LIST may be destroyed, and thus TABLE_LIST
      memory freed.
      
      The fix is to ensure that TABLE_LIST elements
      that are added to the global table list (lex->query_tables)
      are always allocated in thd->mem_root, which is not
      destroyed until end of execution.
      
      If previously TABLE_LIST elements were allocated
      at ha_myisammrg::open() (i.e. when the TABLE
      object was created and added to the table cache),
      now they are allocated in ha_myisammrg::add_chidlren_list()
      (i.e. right after "open" of the merge parent in 
      open_tables()).
      We still create a list of children names
      at ha_myisammrg::open() to use as a basis
      for creation of TABLE_LISTs, that allows
      to avoid reading the merge handler data
      file on every execution.
     @ mysql-test/r/merge_recover.result
        Test results for Bug#46610.
     @ mysql-test/t/merge_recover-master.opt
        Option file for Bug#46610 test (need a new test because
        of that option, which is not tested anywhere else).
     @ mysql-test/t/merge_recover.test
        Add a test case for Bug#46610.
     @ sql/table.h
        MERGE child child_def_version is now moved from TABLE_LIST
        to MERGE engine specific data structure.
     @ storage/myisammrg/ha_myisammrg.h
        Introduce an auxiliary structure to keep MERGE child name
        and definition version. A list of Mrg_child_def is created
        in ha_myisammrg::open() and reused in ha_myisammrg::add_children_list().
[13 Aug 2009 16:14] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/80767

2857 Konstantin Osipov	2009-08-13
      A fix and a test case for Bug#46610 " 	MySQL 5.4.4: MyISAM MRG engine crash 
      on auto-repair of child".
      Also fixes Bug#42862 "Crash on failed attempt to open a children of a 
      merge table".
      
      MERGE engine needs to extend the global table list
      with TABLE_LIST elements for child tables,
      so that they are opened and locked.
      Previously these table list elements were allocated
      in memory of ha_myisammrg object (MERGE engine handler).
      That would lead to access to freed memory in 
      recover_from_failed_open_table_attempt(), which would
      try to recover a MERGE table child (MyISAM table)
      and use for that TABLE_LIST of that child.
      But by the time recover_from_failed_open_table_attempt()
      is invoked, ha_myisammrg object that owns this
      TABLE_LIST may be destroyed, and thus TABLE_LIST
      memory freed.
      
      The fix is to ensure that TABLE_LIST elements
      that are added to the global table list (lex->query_tables)
      are always allocated in thd->mem_root, which is not
      destroyed until end of execution.
      
      If previously TABLE_LIST elements were allocated
      at ha_myisammrg::open() (i.e. when the TABLE
      object was created and added to the table cache),
      now they are allocated in ha_myisammrg::add_chidlren_list()
      (i.e. right after "open" of the merge parent in 
      open_tables()).
      We still create a list of children names
      at ha_myisammrg::open() to use as a basis
      for creation of TABLE_LISTs, that allows
      to avoid reading the merge handler data
      file on every execution.
     @ mysql-test/r/merge_recover.result
        Test results for Bug#46610.
     @ mysql-test/t/merge_recover-master.opt
        Option file for Bug#46610 test (need a new test because
        of that option, which is not tested anywhere else).
     @ mysql-test/t/merge_recover.test
        Add a test case for Bug#46610.
     @ sql/table.h
        MERGE child child_def_version is now moved from TABLE_LIST
        to MERGE engine specific data structure.
     @ storage/myisammrg/ha_myisammrg.h
        Introduce an auxiliary structure to keep MERGE child name
        and definition version. A list of Mrg_child_def is created
        in ha_myisammrg::open() and reused in ha_myisammrg::add_children_list().
[13 Aug 2009 16:16] Konstantin Osipov
Pushed into 5.4.4
[20 Aug 2009 0:17] Paul DuBois
Noted in 5.4.4 changelog.

The server could crash if an attempt to open a MERGE table child
MyISAM table failed.
[24 Aug 2009 13:53] Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090824135126-2rngffvth14a8bpj) (version source revid:kostja@sun.com-20090813161355-0s32oosvtje2u8c6) (merge vers: 5.4.4-alpha) (pib:11)
[8 Dec 2009 13:58] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/93196

3001 Konstantin Osipov	2009-12-08
      Backport of revid 2617.69.21, 2617.69.22, 2617.29.23:
      ----------------------------------------------------------
      revno: 2617.69.21
      committer: Konstantin Osipov <kostja@sun.com>
      branch nick: 5.4-4284-1-assert
      timestamp: Thu 2009-08-13 20:13:55 +0400
      message:
        A fix and a test case for Bug#46610 "MySQL 5.4.4: MyISAM MRG engine crash
        on auto-repair of child".
        Also fixes Bug#42862 "Crash on failed attempt to open a children of a
        merge table".
      
        MERGE engine needs to extend the global table list
        with TABLE_LIST elements for child tables,
        so that they are opened and locked.
        Previously these table list elements were allocated
        in memory of ha_myisammrg object (MERGE engine handler).
        That would lead to access to freed memory in
        recover_from_failed_open_table_attempt(), which would
        try to recover a MERGE table child (MyISAM table)
        and use for that TABLE_LIST of that child.
        But by the time recover_from_failed_open_table_attempt()
        is invoked, ha_myisammrg object that owns this
        TABLE_LIST may be destroyed, and thus TABLE_LIST
        memory freed.
      
        The fix is to ensure that TABLE_LIST elements
        that are added to the global table list (lex->query_tables)
        are always allocated in thd->mem_root, which is not
        destroyed until end of execution.
      
        If previously TABLE_LIST elements were allocated
        at ha_myisammrg::open() (i.e. when the TABLE
        object was created and added to the table cache),
        now they are allocated in ha_myisammrg::add_chidlren_list()
        (i.e. right after "open" of the merge parent in
        open_tables()).
        We still create a list of children names
        at ha_myisammrg::open() to use as a basis
        for creation of TABLE_LISTs, that allows
        to avoid reading the merge handler data
        file on every execution.
     @ mysql-test/r/merge_recover.result
        Test results for Bug#46610.
     @ mysql-test/t/merge_recover-master.opt
        Option file for Bug#46610 test (need a new test because
        of that option, which is not tested anywhere else).
     @ mysql-test/t/merge_recover.test
        Add a test case for Bug#46610.
     @ sql/table.h
        MERGE child child_def_version is now moved from TABLE_LIST
        to MERGE engine specific data structure.
     @ storage/myisammrg/ha_myisammrg.cc
        Introduce an auxiliary structure to keep MERGE child name
        and definition version. A list of Mrg_child_def is created
        in ha_myisammrg::open() and reused in ha_myisammrg::add_children_list().
[16 Feb 2010 16:45] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100216101445-2ofzkh48aq2e0e8o) (version source revid:kostja@sun.com-20091211154405-c9yhiewr9o5d20rq) (merge vers: 6.0.14-alpha) (pib:16)
[16 Feb 2010 16:55] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100216101208-33qkfwdr0tep3pf2) (version source revid:kostja@sun.com-20091208135725-r4s4o5rci1pnp2j8) (pib:16)
[17 Feb 2010 0:35] Paul DuBois
Noted in 6.0.14 changelog.

Setting report to Need Merge pending push of Celosia into release tree.
[6 Mar 2010 11:08] Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:vvaintroub@mysql.com-20100216221947-luyhph0txl2c5tc8) (merge vers: 5.5.99-m3) (pib:16)
[7 Mar 2010 12:55] Paul DuBois
Noted in 5.5.3 changelog.