| Bug #42862 | Crash on failed attempt to open a children of a merge table | ||
|---|---|---|---|
| Submitted: | 14 Feb 2009 20:35 | Modified: | 7 Mar 2010 12:55 |
| Reporter: | Davi Arnaut (OCA) | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Merge storage engine | Severity: | S2 (Serious) |
| Version: | 6.0-bugteam | OS: | Any |
| Assigned to: | Konstantin Osipov | CPU Architecture: | Any |
[15 Feb 2009 8:00]
MySQL Verification Team
Davi, do you think this is the same as bug #41212 ?
[15 Feb 2009 9:07]
Sveta Smirnova
Thank you for the report. I can not repeat described behavior: mysqltest: At line 22: query 'reap' failed: 1168: Unable to open underlying table which is differently defined or of non-MyISAM type or doesn't exist The result from queries just before the failure was: create table t1 (a int not null) engine=csv; create table t2 (a int not null) engine=csv; create table t3 (a int not null) engine=MERGE UNION=(t1,t2); flush tables; set debug_sync='after_close_thread_tables SIGNAL flush WAIT_FOR flushed'; select * from t3;; set debug_sync= 'now WAIT_FOR flush'; Warnings: Warning 1728 debug sync point wait timed out flush tables; set debug_sync= 'now SIGNAL flushed'; Probably test case should be corrected?
[15 Feb 2009 13:50]
Davi Arnaut
The root cause of this crash might be similar to Bug#41212 Sorry for the old test case. This one should yield a crash using latest 6.0-bugteam: --source include/have_csv.inc connect (con1,localhost,root,,); connect (con2,localhost,root,,); create table t1 (a int not null) engine=csv; create table t2 (a int not null) engine=csv; create table t3 (a int not null) engine=MERGE UNION=(t1,t2); connection con1; flush tables; let $MYSQLD_DATADIR= `select @@datadir`; --remove_file $MYSQLD_DATADIR/test/t2.CSM set debug_sync='after_close_thread_tables SIGNAL flush WAIT_FOR flushed'; --send select * from t3; connection con2; set debug_sync= 'now WAIT_FOR flush'; flush tables; set debug_sync= 'now SIGNAL flushed'; connection con1; --reap
[15 Feb 2009 18:40]
Valeriy Kravchuk
Same result as Sveta got, with latest 6.0.10 from bzr: openxs@suse:/home2/openxs/dbs/6.0/mysql-test> ./mysql-test-run.pl bug42862 Logging: ./mysql-test-run.pl bug42862 090405 15:07:02 [Warning] Forcing shutdown of 2 plugins MySQL Version 6.0.10 Checking supported features... - using ndbcluster when necessary, mysqld supports it - SSL connections supported - binaries are debug compiled Collecting tests... vardir: /home2/openxs/dbs/6.0/mysql-test/var Checking leftover processes... Removing old var directory... Creating var directory '/home2/openxs/dbs/6.0/mysql-test/var'... Installing system database... Using server port 1025 worker[1] Using MTR_BUILD_THREAD 250, with reserved ports 12500..12509 ============================================================================== TEST RESULT TIME (ms) ------------------------------------------------------------ main.bug42862 [ fail ] Test ended at 2009-04-05 15:12:26 CURRENT_TEST: main.bug42862 mysqltest: At line 23: query 'reap' failed: 1168: Unable to open underlying table which is differently defined or of non-MyISAM type or doesn't exist The result from queries just before the failure was: create table t1 (a int not null) engine=csv; create table t2 (a int not null) engine=csv; create table t3 (a int not null) engine=MERGE UNION=(t1,t2); flush tables; set debug_sync='after_close_thread_tables SIGNAL flush WAIT_FOR flushed'; select * from t3;; set debug_sync= 'now WAIT_FOR flush'; Warnings: Warning 1728 debug sync point wait timed out flush tables; set debug_sync= 'now SIGNAL flushed'; Warnings from just before the error: Error 1194 Table 't2' is marked as crashed and should be repaired - saving '/home2/openxs/dbs/6.0/mysql-test/var/log/main.bug42862/' to '/home2/openxs/dbs/6.0/mysql-test/var/log/main.bug42862/' ------------------------------------------------------------ The servers were restarted 0 times Spent 0.000 of 326 seconds executing testcases Failed 1/1 tests, 0.00% were successful. Failing test(s): main.bug42862 The log files in var/log may give you some hint of what went wrong. If you want to report this error, please read first the documentation at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html mysql-test-run: *** ERROR: there were failing test cases
[16 Feb 2009 10:07]
Sveta Smirnova
Same for me: can not repeat crash with new test case either.
[16 Feb 2009 10:46]
Sveta Smirnova
Thank you for the feedback. Crash verified with version mysql-6.0-bugteam. With main tree bug is not repeatable.
[16 Feb 2009 10:47]
Sveta Smirnova
Backtrace: #0 0x002ce402 in __kernel_vsyscall () #0 0x002ce402 in __kernel_vsyscall () #1 0x0046264f in pthread_kill () from /lib/libpthread.so.0 #2 0x0881050f in my_write_core (sig=11) at stacktrace.c:309 #3 0x082bab54 in handle_segfault (sig=11) at mysqld.cc:2689 #4 <signal handler called> #5 0x082b599b in mdl_set_lock_type (lock_data=0x8f8f8f8f, lock_type=MDL_EXCLUSIVE) at mdl.h:169 #6 0x08324945 in recover_from_failed_open_table_attempt (thd=0x9a3d600, table=0x9a864c0, action=OT_REPAIR) at sql_base.cc:3515 #7 0x08326a07 in open_tables (thd=0x9a3d600, start=0x9cb75c64, counter=0x9cb75c50, flags=0) at sql_base.cc:3770 #8 0x08327061 in open_and_lock_tables_derived (thd=0x9a3d600, tables=0x9a80b80, derived=true, flags=0) at sql_base.cc:4189 #9 0x082db096 in open_and_lock_tables (thd=0x9a3d600, tables=0x9a80b80) at ../../sql/mysql_priv.h:1570 #10 0x082c9a91 in execute_sqlcom_select (thd=0x9a3d600, all_tables=0x9a80b80) at sql_parse.cc:4722 #11 0x082cf816 in mysql_execute_command (thd=0x9a3d600) at sql_parse.cc:2063 #12 0x082d8669 in mysql_parse (thd=0x9a3d600, inBuf=0x9a80818 "select * from t3", length=16, found_semicolon=0x9cb76f20) at sql_parse.cc:5752 #13 0x082d90ab in dispatch_command (command=COM_QUERY, thd=0x9a3d600, packet=0x9a748f9 "select * from t3;", packet_length=17) at sql_parse.cc:1009 #14 0x082da40f in do_command (thd=0x9a3d600) at sql_parse.cc:691 #15 0x082c82b7 in handle_one_connection (arg=0x9a3d600) at sql_connect.cc:1146 #16 0x0045fbd4 in start_thread () from /lib/libpthread.so.0 #17 0x003b74fe in clone () from /lib/libc.so.6
[16 Feb 2009 11:09]
Davi Arnaut
Crashes on 6.0 main for me too, but crashing on my machine and not on others is essentially due to the undefined behavior nature of invalid access to memory. Valgrind run of the test case will probably show up warnings if it does not crash.
[7 Aug 2009 18:55]
Konstantin Osipov
See also Bug#46610
[13 Aug 2009 15:16]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/80766 2857 Konstantin Osipov 2009-08-13 A fix and a test case for Bug#46610 " MySQL 5.4.4: MyISAM MRG engine crash on auto-repair of child". Also fixes Bug#42862 "Crash on failed attempt to open a children of a merge table". MERGE engine needs to extend the global table list with TABLE_LIST elements for child tables, so that they are opened and locked. Previously these table list elements were allocated in memory of ha_myisammrg object (MERGE engine handler). That would lead to access to freed memory in recover_from_failed_open_table_attempt(), which would try to recover a MERGE table child (MyISAM table) and use for that TABLE_LIST of that child. But by the time recover_from_failed_open_table_attempt() is invoked, ha_myisammrg object that owns this TABLE_LIST may be destroyed, and thus TABLE_LIST memory freed. The fix is to ensure that TABLE_LIST elements that are added to the global table list (lex->query_tables) are always allocated in thd->mem_root, which is not destroyed until end of execution. If previously TABLE_LIST elements were allocated at ha_myisammrg::open() (i.e. when the TABLE object was created and added to the table cache), now they are allocated in ha_myisammrg::add_chidlren_list() (i.e. right after "open" of the merge parent in open_tables()). We still create a list of children names at ha_myisammrg::open() to use as a basis for creation of TABLE_LISTs, that allows to avoid reading the merge handler data file on every execution. @ mysql-test/r/merge_recover.result Test results for Bug#46610. @ mysql-test/t/merge_recover-master.opt Option file for Bug#46610 test (need a new test because of that option, which is not tested anywhere else). @ mysql-test/t/merge_recover.test Add a test case for Bug#46610. @ sql/table.h MERGE child child_def_version is now moved from TABLE_LIST to MERGE engine specific data structure. @ storage/myisammrg/ha_myisammrg.h Introduce an auxiliary structure to keep MERGE child name and definition version. A list of Mrg_child_def is created in ha_myisammrg::open() and reused in ha_myisammrg::add_children_list().
[13 Aug 2009 16:14]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/80767 2857 Konstantin Osipov 2009-08-13 A fix and a test case for Bug#46610 " MySQL 5.4.4: MyISAM MRG engine crash on auto-repair of child". Also fixes Bug#42862 "Crash on failed attempt to open a children of a merge table". MERGE engine needs to extend the global table list with TABLE_LIST elements for child tables, so that they are opened and locked. Previously these table list elements were allocated in memory of ha_myisammrg object (MERGE engine handler). That would lead to access to freed memory in recover_from_failed_open_table_attempt(), which would try to recover a MERGE table child (MyISAM table) and use for that TABLE_LIST of that child. But by the time recover_from_failed_open_table_attempt() is invoked, ha_myisammrg object that owns this TABLE_LIST may be destroyed, and thus TABLE_LIST memory freed. The fix is to ensure that TABLE_LIST elements that are added to the global table list (lex->query_tables) are always allocated in thd->mem_root, which is not destroyed until end of execution. If previously TABLE_LIST elements were allocated at ha_myisammrg::open() (i.e. when the TABLE object was created and added to the table cache), now they are allocated in ha_myisammrg::add_chidlren_list() (i.e. right after "open" of the merge parent in open_tables()). We still create a list of children names at ha_myisammrg::open() to use as a basis for creation of TABLE_LISTs, that allows to avoid reading the merge handler data file on every execution. @ mysql-test/r/merge_recover.result Test results for Bug#46610. @ mysql-test/t/merge_recover-master.opt Option file for Bug#46610 test (need a new test because of that option, which is not tested anywhere else). @ mysql-test/t/merge_recover.test Add a test case for Bug#46610. @ sql/table.h MERGE child child_def_version is now moved from TABLE_LIST to MERGE engine specific data structure. @ storage/myisammrg/ha_myisammrg.h Introduce an auxiliary structure to keep MERGE child name and definition version. A list of Mrg_child_def is created in ha_myisammrg::open() and reused in ha_myisammrg::add_children_list().
[13 Aug 2009 16:16]
Konstantin Osipov
Pushed into 5.4.4
[20 Aug 2009 0:17]
Paul DuBois
Noted in 5.4.4 changelog. The server could crash if an attempt to open a MERGE table child MyISAM table failed.
[24 Aug 2009 13:53]
Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090824135126-2rngffvth14a8bpj) (version source revid:kostja@sun.com-20090813161355-0s32oosvtje2u8c6) (merge vers: 5.4.4-alpha) (pib:11)
[8 Dec 2009 13:58]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/93196 3001 Konstantin Osipov 2009-12-08 Backport of revid 2617.69.21, 2617.69.22, 2617.29.23: ---------------------------------------------------------- revno: 2617.69.21 committer: Konstantin Osipov <kostja@sun.com> branch nick: 5.4-4284-1-assert timestamp: Thu 2009-08-13 20:13:55 +0400 message: A fix and a test case for Bug#46610 "MySQL 5.4.4: MyISAM MRG engine crash on auto-repair of child". Also fixes Bug#42862 "Crash on failed attempt to open a children of a merge table". MERGE engine needs to extend the global table list with TABLE_LIST elements for child tables, so that they are opened and locked. Previously these table list elements were allocated in memory of ha_myisammrg object (MERGE engine handler). That would lead to access to freed memory in recover_from_failed_open_table_attempt(), which would try to recover a MERGE table child (MyISAM table) and use for that TABLE_LIST of that child. But by the time recover_from_failed_open_table_attempt() is invoked, ha_myisammrg object that owns this TABLE_LIST may be destroyed, and thus TABLE_LIST memory freed. The fix is to ensure that TABLE_LIST elements that are added to the global table list (lex->query_tables) are always allocated in thd->mem_root, which is not destroyed until end of execution. If previously TABLE_LIST elements were allocated at ha_myisammrg::open() (i.e. when the TABLE object was created and added to the table cache), now they are allocated in ha_myisammrg::add_chidlren_list() (i.e. right after "open" of the merge parent in open_tables()). We still create a list of children names at ha_myisammrg::open() to use as a basis for creation of TABLE_LISTs, that allows to avoid reading the merge handler data file on every execution. @ mysql-test/r/merge_recover.result Test results for Bug#46610. @ mysql-test/t/merge_recover-master.opt Option file for Bug#46610 test (need a new test because of that option, which is not tested anywhere else). @ mysql-test/t/merge_recover.test Add a test case for Bug#46610. @ sql/table.h MERGE child child_def_version is now moved from TABLE_LIST to MERGE engine specific data structure. @ storage/myisammrg/ha_myisammrg.cc Introduce an auxiliary structure to keep MERGE child name and definition version. A list of Mrg_child_def is created in ha_myisammrg::open() and reused in ha_myisammrg::add_children_list().
[16 Feb 2010 16:45]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100216101445-2ofzkh48aq2e0e8o) (version source revid:kostja@sun.com-20091211154405-c9yhiewr9o5d20rq) (merge vers: 6.0.14-alpha) (pib:16)
[16 Feb 2010 16:55]
Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100216101208-33qkfwdr0tep3pf2) (version source revid:kostja@sun.com-20091208135725-r4s4o5rci1pnp2j8) (pib:16)
[17 Feb 2010 0:35]
Paul DuBois
Noted in 6.0.14 changelog. Setting report to Need Merge pending push of Celosia into release tree.
[6 Mar 2010 11:08]
Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:vvaintroub@mysql.com-20100216221947-luyhph0txl2c5tc8) (merge vers: 5.5.99-m3) (pib:16)
[7 Mar 2010 12:55]
Paul DuBois
Noted in 5.5.3 changelog.
Description: There is a comment in sql_base.cc close_tables_for_reopen(thd, start, (action == OT_BACK_OFF_AND_RETRY)); /* Here we rely on the fact that 'tables' still points to the valid TABLE_LIST element. Altough currently this assumption is valid it may change in future. */ DEBUG_SYNC(thd, "after_close_thread_tables"); if (recover_from_failed_open_table_attempt(thd, tables, action)) Future has came.. It actually does not point to a valid TABLE_LIST if the failed open was for a children of a merge table and a flush tables sneaks between the close tables and recover. How to repeat: Test case: connect (con1,localhost,root,,); connect (con2,localhost,root,,); create table t1 (a int not null) engine=csv; create table t2 (a int not null) engine=csv; create table t3 (a int not null) engine=MERGE UNION=(t1,t2); --connection con1 flush tables; --remove_file $MYSQLTEST_VARDIR/master-data/test/t2.CSM set debug_sync='after_close_thread_tables SIGNAL flush WAIT_FOR flushed'; --send select * from t3; --connection con2 set debug_sync= 'now WAIT_FOR flush'; flush tables; set debug_sync= 'now SIGNAL flushed'; --conneciton con1 --reap Sync point: === modified file 'sql/sql_base.cc' --- sql/sql_base.cc 2008-11-06 18:39:27 +0000 +++ sql/sql_base.cc 2008-11-18 14:50:37 +0000 @@ -3744,6 +3744,7 @@ int open_tables(THD *thd, TABLE_LIST **s TABLE_LIST element. Altough currently this assumption is valid it may change in future. */ + DEBUG_SYNC(thd, "after_close_thread_tables"); if (recover_from_failed_open_table_attempt(thd, tables, action)) { result= -1;