Bug #39674 On shutdown mdl_destroy() called before plugin_shutdown()
Submitted: 26 Sep 2008 15:13 Modified: 7 Mar 2010 18:35
Reporter: Paul McCullagh (Basic Quality Contributor) (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:6.0.6,5.4 OS:Any
Assigned to: Dmitry Lenev CPU Architecture:Any
Tags: crash, mdl, mdl_destroy, plugin_shutdown, qc

[26 Sep 2008 15:13] Paul McCullagh
Description:
Plugins that rely on data that uses the "meta data lock", will not work because the MDL system is shutdown before plugin_shutdown() is called.

In a debug version

In a non-debug version the server hangs on shutdown

How to repeat:
I repeated this error with the PBXT 1.0.05 pluggable storage engine.

You need a debug build of MySQL 6.0.6. 

Download the engine from PBXT from launchpad (do not place it in the storage directory):

bzr branch lp:pbxt/1.0.05-beta pbxt
cd pbxt
./configure --with-mysql=<path-to_mysql-tree>
make install

Start the server and enter:
install plugin PBXT soname 'libpbxt.so';

The error occurs on shutdown if you comment out the following code:

In MySQL:

file sql_plugin.cc line 1392:
    //free_root(tmp_root, MYF(MY_MARK_BLOCKS_FREE));

This is required because Bug #39673 causes the server to crash before the error can be reproduced.

In PBXT:

file myxt_xt.cc line 2700:
	//if (!pbxt_inited)
	//	mdl_init();

This is a hack to avoid the error!

If you are running a debug MySQL server abort() will be called and you will get the following stack trace:

#0  0x920a82ee in __semwait_signal_nocancel ()
#1  0x920a7ed2 in nanosleep$NOCANCEL$UNIX2003 ()
#2  0x9209a1b7 in usleep$NOCANCEL$UNIX2003 ()
#3  0x920bb9bb in abort ()
#4  0x00669863 in safe_mutex_lock (mp=0x9cb1e0, try_lock=0 '\0', file=0x76189a "mdl.cc", line=1317) at thr_mutex.c:107
#5  0x002fa7e0 in mdl_release_locks (context=0x2046c8c) at mdl.cc:1317
#6  0x0014ba82 in close_thread_tables (thd=0x2046c18, skip_mdl=false) at sql_base.cc:1433
#7  0x01b276b2 in myxt_destroy_thread (thread=0x2046c18) at myxt_xt.cc:2702
#8  0x01b38543 in tabc_fr_run_thread (self=0x2044608) at tabcache_xt.cc:1112
#9  0x01b49c10 in thr_main (data=0xbfffe5cc) at thread_xt.cc:1010
#10 0x91ff7c55 in _pthread_start ()
#11 0x91ff7b12 in thread_start ()

Suggested fix:
On shutdown, call mdl_destroy() after the call to plugin_shutdown().
[20 Feb 2009 10:28] Sergei Golubchik
Paul, why do you need to call close_thread_tables() from your plugin shutdown function ?
[20 Feb 2009 10:52] Paul McCullagh
Hi Sergei,

I have some background threads which need to call into MySQL to open tables. For this purpose they need a THD.

So on startup of the background threads I create a THD, and on shutdown I destroy the THD.

Before a THD is deleted I am required to call close_thread_tables(thd) to cleanup.
[25 Feb 2009 11:37] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/67518

2728 Dmitry Lenev	2009-02-25
      Fix for bug #39674 "On shutdown mdl_destroy() called before
      plugin_shutdown()".
      
      Attempt to shutdown PBXT engine plugin led to assertion failure 
      caused by using already destroyed mutex in metadata locking 
      subsystem.
      
      This problem stemmed from the fact that we MDL subsystem and
      table definition cache were deinitialized before plugin shutdown 
      while PBXT plugin during its shutdown process accessed tables and 
      therefore expected them to be in working shape.
      
      This patch solves this problem by moving deinitialization of
      these two subsystems after plugins are shut down.
      
      No test case is provided since such test case would require
      using PBXT or other plugin which accesses tables during its 
      shutdown process.
[24 Jul 2009 20:29] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/79285

2830 Dmitry Lenev	2009-07-25
      Fix for bug #39674 "On shutdown mdl_destroy() called before
      plugin_shutdown()".
      
      Attempt to shutdown PBXT engine plugin led to assertion failure
      caused by using already destroyed mutex in metadata locking
      subsystem.
      
      This problem stemmed from the fact that we MDL subsystem and
      table definition cache were deinitialized before plugin shutdown
      while PBXT plugin during its shutdown process accessed tables and
      therefore expected them to be in working shape.
      
      This patch solves this problem by moving deinitialization of
      these two subsystems after plugins are shut down.
      
      No test case is provided since such test case would require
      using PBXT or other plugin which accesses tables during its
      shutdown process.
     @ sql/mysql_priv.h
        Introduced table_def_start_shutdown() function which informs
        table definition cache that shutdown process has been started
        so it has to keep number of TABLE and TABLE_SHARE objects minimal
        in order to reduce number of references to pluggable engines.
     @ sql/mysqld.cc
        Destroy table definition cache and meta-data locking subsystem
        after shutting down plugins. This allows plugins to work with
        tables during their shutdown.
        Since table definition cache hold references to storage engine
        plugins we have to remove unused tables from it before shutting
        down plugins and keep number of these references minimal during
        the process (by immediately removing tables opened during this
        process from the table definition cache).
     @ sql/sql_base.cc
        Introduced table_def_start_shutdown() function which informs
        table definition cache that shutdown process has been started
        so it has to keep number of TABLE and TABLE_SHARE objects
        minimal in order to reduce number of references to pluggable
        engines.
        This allows to smoothly shutdown such plugins without completely
        prohibiting access to tables/table definition cache while
        shutting down other plugins.
[25 Jul 2009 8:01] Dmitry Lenev
Patch for this bug was queued into mysql-azalea-bugfixing tree.
[4 Aug 2009 19:50] Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090804194615-h40sa098mx4z49qg) (version source revid:iggy@mysql.com-20090731204544-7nio1afvg0dmzs7g) (merge vers: 5.4.4-alpha) (pib:11)
[7 Aug 2009 0:39] Paul DuBois
Noted in 5.4.4 changelog.

Plugin shutdown could lead to an assertion failure caused by using an
already destroyed mutex in the metadata locking subsystem.
[12 Aug 2009 22:50] Paul DuBois
Noted in 5.4.2 changelog because next 5.4 version will be 5.4.2 and not 5.4.4.
[15 Aug 2009 2:07] Paul DuBois
Ignore previous comment about 5.4.2.
[8 Dec 2009 8:27] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/93118

2998 Konstantin Osipov	2009-12-08
      Backport of:
      ------------------------------------------------------------
      revno: 2617.65.6
      committer: Dmitry Lenev <dlenev@mysql.com>
      branch nick: mysql-azalea-bg39674
      timestamp: Sat 2009-07-25 00:28:43 +0400
      message:
        Fix for bug #39674 "On shutdown mdl_destroy() called before
        plugin_shutdown()".
      
        Attempt to shutdown PBXT engine plugin led to assertion failure
        caused by using already destroyed mutex in metadata locking
        subsystem.
      
        This problem stemmed from the fact that we MDL subsystem and
        table definition cache were deinitialized before plugin shutdown
        while PBXT plugin during its shutdown process accessed tables and
        therefore expected them to be in working shape.
      
        This patch solves this problem by moving deinitialization of
        these two subsystems after plugins are shut down.
      
        No test case is provided since such test case would require
        using PBXT or other plugin which accesses tables during its
        shutdown process.
     @ sql/mysql_priv.h
        Introduced table_def_start_shutdown() function which informs
        table definition cache that shutdown process has been started
        so it has to keep number of TABLE and TABLE_SHARE objects minimal
        in order to reduce number of references to pluggable engines.
     @ sql/mysqld.cc
        Destroy table definition cache and meta-data locking subsystem
        after shutting down plugins. This allows plugins to work with
        tables during their shutdown.
        Since table definition cache hold references to storage engine
        plugins we have to remove unused tables from it before shutting
        down plugins and keep number of these references minimal during
        the process (by immediately removing tables opened during this
        process from the table definition cache).
     @ sql/sql_base.cc
        Introduced table_def_start_shutdown() function which informs
        table definition cache that shutdown process has been started
        so it has to keep number of TABLE and TABLE_SHARE objects
        minimal in order to reduce number of references to pluggable
        engines.
                This allows to smoothly shutdown such plugins without completely
                prohibiting access to tables/table definition cache while
                shutting down other plugins.
[16 Feb 2010 16:46] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100216101445-2ofzkh48aq2e0e8o) (version source revid:kostja@sun.com-20091211154405-c9yhiewr9o5d20rq) (merge vers: 6.0.14-alpha) (pib:16)
[16 Feb 2010 16:55] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100216101208-33qkfwdr0tep3pf2) (version source revid:kostja@sun.com-20091208082649-mfiuec3c36xwrxw3) (pib:16)
[6 Mar 2010 11:06] Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:vvaintroub@mysql.com-20100216221947-luyhph0txl2c5tc8) (merge vers: 5.5.99-m3) (pib:16)
[7 Mar 2010 18:35] Paul DuBois
Noted in 5.5.3, 6.0.14 changelogs.