| Bug #39674 | On shutdown mdl_destroy() called before plugin_shutdown() | ||
|---|---|---|---|
| Submitted: | 26 Sep 2008 15:13 | Modified: | 7 Mar 2010 18:35 |
| Reporter: | Paul McCullagh (Basic Quality Contributor) (OCA) | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server | Severity: | S2 (Serious) |
| Version: | 6.0.6,5.4 | OS: | Any |
| Assigned to: | Dmitry Lenev | CPU Architecture: | Any |
| Tags: | crash, mdl, mdl_destroy, plugin_shutdown, qc | ||
[20 Feb 2009 10:28]
Sergei Golubchik
Paul, why do you need to call close_thread_tables() from your plugin shutdown function ?
[20 Feb 2009 10:52]
Paul McCullagh
Hi Sergei, I have some background threads which need to call into MySQL to open tables. For this purpose they need a THD. So on startup of the background threads I create a THD, and on shutdown I destroy the THD. Before a THD is deleted I am required to call close_thread_tables(thd) to cleanup.
[25 Feb 2009 11:37]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/67518 2728 Dmitry Lenev 2009-02-25 Fix for bug #39674 "On shutdown mdl_destroy() called before plugin_shutdown()". Attempt to shutdown PBXT engine plugin led to assertion failure caused by using already destroyed mutex in metadata locking subsystem. This problem stemmed from the fact that we MDL subsystem and table definition cache were deinitialized before plugin shutdown while PBXT plugin during its shutdown process accessed tables and therefore expected them to be in working shape. This patch solves this problem by moving deinitialization of these two subsystems after plugins are shut down. No test case is provided since such test case would require using PBXT or other plugin which accesses tables during its shutdown process.
[24 Jul 2009 20:29]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/79285 2830 Dmitry Lenev 2009-07-25 Fix for bug #39674 "On shutdown mdl_destroy() called before plugin_shutdown()". Attempt to shutdown PBXT engine plugin led to assertion failure caused by using already destroyed mutex in metadata locking subsystem. This problem stemmed from the fact that we MDL subsystem and table definition cache were deinitialized before plugin shutdown while PBXT plugin during its shutdown process accessed tables and therefore expected them to be in working shape. This patch solves this problem by moving deinitialization of these two subsystems after plugins are shut down. No test case is provided since such test case would require using PBXT or other plugin which accesses tables during its shutdown process. @ sql/mysql_priv.h Introduced table_def_start_shutdown() function which informs table definition cache that shutdown process has been started so it has to keep number of TABLE and TABLE_SHARE objects minimal in order to reduce number of references to pluggable engines. @ sql/mysqld.cc Destroy table definition cache and meta-data locking subsystem after shutting down plugins. This allows plugins to work with tables during their shutdown. Since table definition cache hold references to storage engine plugins we have to remove unused tables from it before shutting down plugins and keep number of these references minimal during the process (by immediately removing tables opened during this process from the table definition cache). @ sql/sql_base.cc Introduced table_def_start_shutdown() function which informs table definition cache that shutdown process has been started so it has to keep number of TABLE and TABLE_SHARE objects minimal in order to reduce number of references to pluggable engines. This allows to smoothly shutdown such plugins without completely prohibiting access to tables/table definition cache while shutting down other plugins.
[25 Jul 2009 8:01]
Dmitry Lenev
Patch for this bug was queued into mysql-azalea-bugfixing tree.
[4 Aug 2009 19:50]
Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090804194615-h40sa098mx4z49qg) (version source revid:iggy@mysql.com-20090731204544-7nio1afvg0dmzs7g) (merge vers: 5.4.4-alpha) (pib:11)
[7 Aug 2009 0:39]
Paul DuBois
Noted in 5.4.4 changelog. Plugin shutdown could lead to an assertion failure caused by using an already destroyed mutex in the metadata locking subsystem.
[12 Aug 2009 22:50]
Paul DuBois
Noted in 5.4.2 changelog because next 5.4 version will be 5.4.2 and not 5.4.4.
[15 Aug 2009 2:07]
Paul DuBois
Ignore previous comment about 5.4.2.
[8 Dec 2009 8:27]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/93118 2998 Konstantin Osipov 2009-12-08 Backport of: ------------------------------------------------------------ revno: 2617.65.6 committer: Dmitry Lenev <dlenev@mysql.com> branch nick: mysql-azalea-bg39674 timestamp: Sat 2009-07-25 00:28:43 +0400 message: Fix for bug #39674 "On shutdown mdl_destroy() called before plugin_shutdown()". Attempt to shutdown PBXT engine plugin led to assertion failure caused by using already destroyed mutex in metadata locking subsystem. This problem stemmed from the fact that we MDL subsystem and table definition cache were deinitialized before plugin shutdown while PBXT plugin during its shutdown process accessed tables and therefore expected them to be in working shape. This patch solves this problem by moving deinitialization of these two subsystems after plugins are shut down. No test case is provided since such test case would require using PBXT or other plugin which accesses tables during its shutdown process. @ sql/mysql_priv.h Introduced table_def_start_shutdown() function which informs table definition cache that shutdown process has been started so it has to keep number of TABLE and TABLE_SHARE objects minimal in order to reduce number of references to pluggable engines. @ sql/mysqld.cc Destroy table definition cache and meta-data locking subsystem after shutting down plugins. This allows plugins to work with tables during their shutdown. Since table definition cache hold references to storage engine plugins we have to remove unused tables from it before shutting down plugins and keep number of these references minimal during the process (by immediately removing tables opened during this process from the table definition cache). @ sql/sql_base.cc Introduced table_def_start_shutdown() function which informs table definition cache that shutdown process has been started so it has to keep number of TABLE and TABLE_SHARE objects minimal in order to reduce number of references to pluggable engines. This allows to smoothly shutdown such plugins without completely prohibiting access to tables/table definition cache while shutting down other plugins.
[16 Feb 2010 16:46]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100216101445-2ofzkh48aq2e0e8o) (version source revid:kostja@sun.com-20091211154405-c9yhiewr9o5d20rq) (merge vers: 6.0.14-alpha) (pib:16)
[16 Feb 2010 16:55]
Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100216101208-33qkfwdr0tep3pf2) (version source revid:kostja@sun.com-20091208082649-mfiuec3c36xwrxw3) (pib:16)
[6 Mar 2010 11:06]
Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:vvaintroub@mysql.com-20100216221947-luyhph0txl2c5tc8) (merge vers: 5.5.99-m3) (pib:16)
[7 Mar 2010 18:35]
Paul DuBois
Noted in 5.5.3, 6.0.14 changelogs.
Description: Plugins that rely on data that uses the "meta data lock", will not work because the MDL system is shutdown before plugin_shutdown() is called. In a debug version In a non-debug version the server hangs on shutdown How to repeat: I repeated this error with the PBXT 1.0.05 pluggable storage engine. You need a debug build of MySQL 6.0.6. Download the engine from PBXT from launchpad (do not place it in the storage directory): bzr branch lp:pbxt/1.0.05-beta pbxt cd pbxt ./configure --with-mysql=<path-to_mysql-tree> make install Start the server and enter: install plugin PBXT soname 'libpbxt.so'; The error occurs on shutdown if you comment out the following code: In MySQL: file sql_plugin.cc line 1392: //free_root(tmp_root, MYF(MY_MARK_BLOCKS_FREE)); This is required because Bug #39673 causes the server to crash before the error can be reproduced. In PBXT: file myxt_xt.cc line 2700: //if (!pbxt_inited) // mdl_init(); This is a hack to avoid the error! If you are running a debug MySQL server abort() will be called and you will get the following stack trace: #0 0x920a82ee in __semwait_signal_nocancel () #1 0x920a7ed2 in nanosleep$NOCANCEL$UNIX2003 () #2 0x9209a1b7 in usleep$NOCANCEL$UNIX2003 () #3 0x920bb9bb in abort () #4 0x00669863 in safe_mutex_lock (mp=0x9cb1e0, try_lock=0 '\0', file=0x76189a "mdl.cc", line=1317) at thr_mutex.c:107 #5 0x002fa7e0 in mdl_release_locks (context=0x2046c8c) at mdl.cc:1317 #6 0x0014ba82 in close_thread_tables (thd=0x2046c18, skip_mdl=false) at sql_base.cc:1433 #7 0x01b276b2 in myxt_destroy_thread (thread=0x2046c18) at myxt_xt.cc:2702 #8 0x01b38543 in tabc_fr_run_thread (self=0x2044608) at tabcache_xt.cc:1112 #9 0x01b49c10 in thr_main (data=0xbfffe5cc) at thread_xt.cc:1010 #10 0x91ff7c55 in _pthread_start () #11 0x91ff7b12 in thread_start () Suggested fix: On shutdown, call mdl_destroy() after the call to plugin_shutdown().