Bug #46044 MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY FOR UPDATE
Submitted: 8 Jul 2009 15:38 Modified: 7 Mar 2010 2:08
Reporter: Matthias Leich Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Locking Severity:S3 (Non-critical)
Version:5.4 OS:Any
Assigned to: Dmitry Lenev CPU Architecture:Any
Tags: deadlock, locking, mdl

[8 Jul 2009 15:38] Matthias Leich
Description:
I get a deadlock (= all "worker" sessions wait
for a table or a lock) after a short runtime in RQG.
The CPU consumption is ~ 0%.

Grammar for object creation:

$tables = {
        rows => [ 1, 10 ]
};

$fields = {
        types => [ 'int' ],
        indexes => [undef ]
};

$data = {
        numbers => [ 'null' ],
        temporals => [ undef ],
        strings => [ 'letter' ]
}

Grammar for concurrent SQL:
query:
   # The HIGH_PRIORITY and FOR UPDATE seem to be necessary.
   # Just the SELECT .... without the CREATE TABLE etc. is not sufficient.
   CREATE TABLE my_table AS SELECT HIGH_PRIORITY A . _field FROM _table AS A LEFT JOIN _table AS B USING (`pk`) LIMIT _digit FOR UPDATE; SELECT SLEEP(1); DROP TABLE my_table |
   LOCK TABLE _table READ , _table READ;
   # Locking two tables seems to be necessary

gendata.pl contains direct after the creation of a table
following additional statements which generate a VIEW
per every base table:
my $create_result = output ("CREATE VIEW `v$table->[TABLE_NAME]` AS SELECT * FROM `$table->[TABLE_NAME]`");
if ($create_result > 1) {
   say("# Unable to create view $table->[TABLE_NAME], skipping...");
   next;
}

My command:
----------
perl runall.pl --mem --basedir=/work2/6.0/azalea/ \
     --threads=100 --queries=1000 --debug \
     --mysqld=--table-lock-wait-timeout=5 \
     --mysqld=--innodb-lock-wait-timeout=5 \
     --gendata=conf/ml_bugnnnnn_data.zz \
     --grammar=conf/ml_bugnnnnn_sql.yy --seed=0

State after 1 - 2 minutes:
--------------------------
The growth of general_log.CSV has ended.
SHOW OPEN TABLES FROM test ;
Database        Table   In_use  Name_locked
test    table1_int_autoinc      20      0
test    vtable10_int_autoinc    0       0
test    vtable1_int_autoinc     0       0
test    table10_int_autoinc     20      0

PROCESSLIST:
19 session in state  "Table lock"
81 sessions in state "Waiting for table"

Most probably important/suspicious things:
1. Only MyISAM and no transactional engines or
   partitioning are involved.
   The VIEWs seem to be needed
2. IMHO a CREATE TABLE ... AS SELECT with
   "HIGH_PRIORITY" and "FOR UPDATE" looks somehow
   very unusual. I can at least imagine that
   HIGH_PRIORITY could give sometimes benefits.
   But I am unsure if it is really intended that
   "FOR UPDATE" in such a CREATE TABLE is allowed.
Whenever I tried to remove the "HIGH_PRIORITY" or
the "FOR UPDATE" the deadlock disappeared.

At looks as if the "--table-lock-wait-timeout=5"
has no impact on the outcome of the test at all.
This is a pity because it could at least in theory
break the deadlock.

I will attach a file with backtraces which I got
after kill -11 <server process>.

My environment:
- MySQL azalea last change 2009-07-02
- BUILD/compile-pentium64-debug-max
- Linux OpenSuSE 11.0 (64 Bit)
- Intel Core2Duo

In case you judge that the current grammar is expected
to create deadlocks because of missing UNLOCK tables.
Please use the following grammar
query:
   # The HIGH_PRIORITY and FOR UPDATE seem to be necessary
   CREATE TABLE my_table AS SELECT HIGH_PRIORITY A . _field FROM _table AS A LEFT JOIN _table AS B USING (`pk`) LIMIT _digit FOR UPDATE; SELECT SLEEP(1); DROP TABLE my_table |
   LOCK TABLE _table READ , _table READ; SELECT SLEEP(1) UNLOCK TABLES;
and observe the deadlock.

How to repeat:
Please use the attached grammar files.

Suggested fix:
Please modify the server so that "--table-lock-wait-timeout=5"
resolves this deadlock.
I am aware that especially the CREATE TABLE statement
looks ugly and it is rather rare that somebody else
runs similar stuff.
[8 Jul 2009 16:17] Matthias Leich
Grammar of creation of objects

Attachment: bug46044_data.zz (application/octet-stream, text), 214 bytes.

[8 Jul 2009 16:18] Matthias Leich
Grammar for stress testing

Attachment: bug46044_sql.yy (application/octet-stream, text), 396 bytes.

[8 Jul 2009 16:22] Matthias Leich
Test output + backtrace after kill -11 at the end

Attachment: bug.prt.gz (application/x-gzip, text), 31.75 KiB.

[8 Jul 2009 18:58] Matthias Leich
There is a missing ";" within the alternative grammar.
Correct would be:
query:
   # The HIGH_PRIORITY and FOR UPDATE seem to be necessary
   CREATE TABLE my_table AS SELECT HIGH_PRIORITY A . _field FROM _table AS A LEFT JOIN
_table AS B USING (`pk`) LIMIT _digit FOR UPDATE; SELECT SLEEP(1); DROP TABLE my_table |
   LOCK TABLE _table READ , _table READ; SELECT SLEEP(1); UNLOCK TABLES;

And with this grammar the deadlock disappears.

But this does not change the fact that the outcome of
the test with the first grammar is strange:
1. http://dev.mysql.com/doc/refman/6.0/en/select.html
   ...
   If you use FOR UPDATE with a storage engine that uses
   page or row locks, ...
   We are using MyISAM so this should be irrelevant
   = have no effect.
   But why disappears the deadlock if the "FOR UPDATE"
   is removed.
2. CREATE TABLE commits at start and end of executing the
   CREATE TABLE base operations like store metadata and
   fill the table (If AS SELECT ...).
   So it should UNLOCK tables already locked by the current
   session. Therefore I think this means the missing balance
   "We have a LOCK TABLE but no UNLOCK" should be no problem.
[14 Aug 2009 20:10] Konstantin Osipov
The most apparent offender:

# 16:48:51 Thread 41 (process 5514):
# 16:48:51 #0  0x00007fbf1858bdd9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
# 16:48:51 #1  0x0000000000b30105 in safe_cond_wait (cond=0x12534c0, mp=0x1253400, file=0xd69b20 "mdl.cc", line=1324) at thr_mutex.c:423
# 16:48:51 #2  0x000000000091ba0e in MDL_context::wait_for_locks (this=0x27b20a0) at mdl.cc:1324
# 16:48:51 #3  0x0000000000757cfb in recover_from_failed_open_table_attempt (thd=0x27b1fc8, table=0x27f4660, action=OT_BACK_OFF_AND_RETRY) at sql_base.cc:3516
# 16:48:51 #4  0x0000000000759c75 in open_tables (thd=0x27b1fc8, start=0x448a25f0, counter=0x448a262c, flags=2) at sql_base.cc:3805
# 16:48:51 #5  0x000000000075a285 in open_normal_and_derived_tables (thd=0x27b1fc8, tables=0x27f4660, flags=2) at sql_base.cc:4276
# 16:48:51 #6  0x000000000088433e in fill_schema_show_cols_or_idxs (thd=0x27b1fc8, tables=0x27f5258, schema_table=0x1132160, open_tables_state_backup=0x448a2ab0) at sql_show.cc:2953
# 16:48:51 #7  0x00000000008953a2 in get_all_tables (thd=0x27b1fc8, tables=0x27f5258, cond=0x0) at sql_show.cc:3361
# 16:48:51 #8  0x0000000000880d4a in get_schema_tables_result (join=0x26a8ca0, executed_place=PROCESSED_BY_JOIN_EXEC) at sql_show.cc:6510
# 16:48:51 #9  0x00000000007a082e in JOIN::exec (this=0x26a8ca0) at sql_select.cc:2393
# 16:48:51 #10 0x000000000079d004 in mysql_select (thd=0x27b1fc8, rref_pointer_array=0x27b3f80, tables=0x27f5258, wild_num=0, fields=@0x27b3ea0, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2686732800, 
# 16:48:51     result=0x28328c0, unit=0x27b3930, select_lex=0x27b3d98) at sql_select.cc:3070
# 16:48:51 #11 0x00000000007a2887 in handle_select (thd=0x27b1fc8, lex=0x27b3890, result=0x28328c0, setup_tables_done_option=0) at sql_select.cc:305
# 16:48:51 #12 0x00000000006fdb07 in execute_sqlcom_select (thd=0x27b1fc8, all_tables=0x27f5258) at sql_parse.cc:4976
# 16:48:51 #13 0x00000000006ff359 in mysql_execute_command (thd=0x27b1fc8) at sql_parse.cc:2161
# 16:48:51 #14 0x0000000000707914 in mysql_parse (thd=0x27b1fc8, inBuf=0x27f4070 "SHOW FIELDS FROM my_table", length=25, found_semicolon=0x448a4f30) at sql_parse.cc:5991
# 16:48:51 #15 0x00000000007084fe in dispatch_command (command=COM_QUERY, thd=0x27b1fc8, packet=0x27ceae9 "\n\t\t\tSHOW FIELDS FROM my_table", packet_length=29) at sql_parse.cc:1061
# 16:48:51 #16 0x00000000007099c2 in do_command (thd=0x27b1fc8) at sql_parse.cc:743
# 16:48:51 #17 0x00000000006f7038 in handle_one_connection (arg=0x27b1fc8) at sql_connect.cc:1158
# 16:48:51 #18 0x00007fbf18588040 in start_thread () from /lib64/libpthread.so.0
# 16:48:51 #19 0x00007fbf1752908d in clone () from /lib64/libc.so.6
# 16:48:51 #20 0x0000000000000000 in ?? ()

Matthias, are you sure the bug is not repeatable in 5.1?
[20 Aug 2009 11:40] Dmitry Lenev
After investigating cause behind this problem I was able to come-up with the following simpler and IMO more natural test case for mysqltest tool which exposes the same issue:

create table t1 (i int);

connect (con1, localhost, root,,);
connection default;

lock tables t1 write;

connection con1;

--send create table t2 select * from t1;

connection default;

--sleep 2

--echo # The below statement hangs causing deadlock.
show fields from t2;

connection con1;
--reap
[27 Aug 2009 6:23] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/81655

2802 Dmitry Lenev	2009-08-27
      Fix for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY 
      FOR UPDATE".
      
      Deadlock occured when during execution of query to I_S we tried to open
      a table or its .FRM in order to get information about it and had to wait
      because we have encountered exclusive metadata lock on this table held by 
      a DDL operation from another connection which in its turn waited for some
      resource currently owned by connection executing this I_S query. 
      For example, this might have happened if one under LOCK TABLES executed I_S
      query targeted to particular table (which was not among locked) and also
      concurrently tried to create this table using CREATE TABLE SELECT which
      had to wait for one of tables locked by the first connection.
      Another situation in which deadlock might have occured is when I_S query,
      which was executed as part of transaction, tried to get information about 
      table which just has been dropped by concurrent DROP TABLES executed under 
      LOCK TABLES and this DROP TABLES for its completion also had to wait 
      transaction from the first connection.
      
      This problem stemmed from the fact that opening of tables/.FRMs for I_S 
      filling is happening outside of connection's main MDL_context so code 
      which tries to detect deadlocks due to conflicting metadata locks doesn't 
      work in this case. Indeed, this led to deadlocks when during I_S filling 
      we tried to wait for conflicting metadata lock to go away, while its owner
      was waiting for some resource held by connection executing I_S query.
      
      This patch solves this problem by avoiding waiting in such situation.
      Instead we skip this table and produce warning that information about
      it was omitted from I_S due to concurrent DDL operation. We still wait
      for conflicting metadata lock to go away when it is known that deadlock
      is not possible (i.e. when connection executing I_S query does not hold
      any metadata or table-level locks).
      Basically, we apply our standard deadlock avoidance technique for metadata
      locks to the process of filling of I_S tables but replace ER_LOCK_DEADLOCK
      error with a warning.
      
      Note that this change is supposed to be safe for 'mysqldump' since the
      only its mode which is affected by this change is --single-transaction mode
      is not safe in the presence of concurrent DDL anyway (and this fact is
      documented). Other modes are unaffected because they either use
      SHOW TABLES/SELECT * FROM I_S.TABLE_NAMES which do not take any metadata
      locks in the process of I_S table filling and thus cannot skip tables or
      execute I_S queries for tables which were previously locked by LOCK TABLES
      (or in the presence of global read lock) which excludes possibility of
      encountering conflicting metadata lock.
     @ mysql-test/r/mdl_sync.result
        Added test for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE
        HIGH_PRIORITY FOR UPDATE".
     @ mysql-test/t/mdl_sync.test
        Added test for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE
        HIGH_PRIORITY FOR UPDATE".
     @ sql/mysql_priv.h
        Added a new flag for open_table() call which allows it to fail
        with an error in cases when conflicting metadata lock is discovered
        instead of waiting until this lock goes away.
     @ sql/share/errmsg-utf8.txt
        Added error/warning message to be generated in cases when information
        about table is omitted from I_S since there is conflicting metadata lock
        on the table.
     @ sql/share/errmsg.txt
        Added error/warning message to be generated in cases when information
        about table is omitted from I_S since there is conflicting metadata lock
        on the table.
     @ sql/sql_base.cc
        Added a new flag for open_table() call which allows it to fail
        with an error in cases when conflicting metadata lock is discovered
        instead of waiting until this lock goes away.
     @ sql/sql_show.cc
        When we are opening a table (or just .FRM) in order to fill I_S with
        information about this table and encounter conflicting metadata lock
        waiting for this lock to go away can lead to a deadlock in some
        situations (under LOCK TABLES, within transaction, etc.). To avoid
        these deadlocks we detect such situations and don't do waiting.
        Instead, we skip table for which we have conflicting metadata lock,
        thus omitting information about it from I_S table, and produce an
        appropriate warning.
        
        Also tried to clarify error handling in fill_schema_table_from_frm().
[27 Aug 2009 6:25] Dmitry Lenev
Fix for this bug was queued into mysql-next-bugfixing tree.
[28 Aug 2009 9:53] Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090828095047-jbux7pie1yjnwpce) (version source revid:dlenev@mysql.com-20090827062217-rgx1i5eo3bcrd1v8) (merge vers: 5.4.4-alpha) (pib:11)
[29 Aug 2009 23:28] Paul DuBois
Deadlock could occur for INFORMATION_SCHEMA queries when execution
attempted to open a table or its .frm file.
[29 Aug 2009 23:29] Paul DuBois
Noted in 5.4.4 changelog.
[9 Dec 2009 22:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/93402

3014 Konstantin Osipov	2009-12-09
      Backport of:
      ------------------------------------------------------------
      revno: 2617.68.7
      committer: Dmitry Lenev <dlenev@mysql.com>
      branch nick: mysql-next-bg46044
      timestamp: Thu 2009-08-27 10:22:17 +0400
      message:
        Fix for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY
        FOR UPDATE".
      
        Deadlock occured when during execution of query to I_S we tried to open
        a table or its .FRM in order to get information about it and had to wait
        because we have encountered exclusive metadata lock on this table held by
        a DDL operation from another connection which in its turn waited for some
        resource currently owned by connection executing this I_S query.
        For example, this might have happened if one under LOCK TABLES executed I_S
        query targeted to particular table (which was not among locked) and also
        concurrently tried to create this table using CREATE TABLE SELECT which
        had to wait for one of tables locked by the first connection.
        Another situation in which deadlock might have occured is when I_S query,
        which was executed as part of transaction, tried to get information about
        table which just has been dropped by concurrent DROP TABLES executed under
        LOCK TABLES and this DROP TABLES for its completion also had to wait
        transaction from the first connection.
      
        This problem stemmed from the fact that opening of tables/.FRMs for I_S
        filling is happening outside of connection's main MDL_context so code
        which tries to detect deadlocks due to conflicting metadata locks doesn't
        work in this case. Indeed, this led to deadlocks when during I_S filling
        we tried to wait for conflicting metadata lock to go away, while its owner
        was waiting for some resource held by connection executing I_S query.
      
        This patch solves this problem by avoiding waiting in such situation.
        Instead we skip this table and produce warning that information about
        it was omitted from I_S due to concurrent DDL operation. We still wait
        for conflicting metadata lock to go away when it is known that deadlock
        is not possible (i.e. when connection executing I_S query does not hold
        any metadata or table-level locks).
        Basically, we apply our standard deadlock avoidance technique for metadata
        locks to the process of filling of I_S tables but replace ER_LOCK_DEADLOCK
        error with a warning.
        Note that this change is supposed to be safe for 'mysqldump' since the
        only its mode which is affected by this change is --single-transaction mode
        is not safe in the presence of concurrent DDL anyway (and this fact is
        documented). Other modes are unaffected because they either use
        SHOW TABLES/SELECT * FROM I_S.TABLE_NAMES which do not take any metadata
        locks in the process of I_S table filling and thus cannot skip tables or
        execute I_S queries for tables which were previously locked by LOCK TABLES
        (or in the presence of global read lock) which excludes possibility of
        encountering conflicting metadata lock.
     @ mysql-test/r/mdl_sync.result
        Added test for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE
        HIGH_PRIORITY FOR UPDATE".
     @ mysql-test/t/mdl_sync.test
        Added test for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE
        HIGH_PRIORITY FOR UPDATE".
     @ sql/mysql_priv.h
        Added a new flag for open_table() call which allows it to fail
        with an error in cases when conflicting metadata lock is discovered
        instead of waiting until this lock goes away.
     @ sql/share/errmsg-utf8.txt
        Added error/warning message to be generated in cases when information
        about table is omitted from I_S since there is conflicting metadata lock
        on the table.
     @ sql/share/errmsg.txt
        Added error/warning message to be generated in cases when information
        about table is omitted from I_S since there is conflicting metadata lock
        on the table.
     @ sql/sql_base.cc
        Added a new flag for open_table() call which allows it to fail
        with an error in cases when conflicting metadata lock is discovered
        instead of waiting until this lock goes away.
     @ sql/sql_show.cc
        When we are opening a table (or just .FRM) in order to fill I_S with
        information about this table and encounter conflicting metadata lock
        waiting for this lock to go away can lead to a deadlock in some
        situations (under LOCK TABLES, within transaction, etc.). To avoid
        these deadlocks we detect such situations and don't do waiting.
        Instead, we skip table for which we have conflicting metadata lock,
        thus omitting information about it from I_S table, and produce an
        appropriate warning.
[16 Feb 2010 16:46] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100216101445-2ofzkh48aq2e0e8o) (version source revid:kostja@sun.com-20091211154405-c9yhiewr9o5d20rq) (merge vers: 6.0.14-alpha) (pib:16)
[16 Feb 2010 16:55] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100216101208-33qkfwdr0tep3pf2) (version source revid:kostja@sun.com-20091209154842-k3dh01k9uj8sasz9) (pib:16)
[6 Mar 2010 11:06] Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:vvaintroub@mysql.com-20100216221947-luyhph0txl2c5tc8) (merge vers: 5.5.99-m3) (pib:16)
[7 Mar 2010 2:08] Paul DuBois
Noted in 5.5.3, 6.0.14 changelogs.
[13 Apr 2010 5:03] Paul DuBois
Correction: Not present in any 5.5.x release. 5.5.3 changelog entry removed.