| Bug #46044 | MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY FOR UPDATE | ||
|---|---|---|---|
| Submitted: | 8 Jul 2009 15:38 | Modified: | 7 Mar 2010 2:08 |
| Reporter: | Matthias Leich | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Locking | Severity: | S3 (Non-critical) |
| Version: | 5.4 | OS: | Any |
| Assigned to: | Dmitry Lenev | CPU Architecture: | Any |
| Tags: | deadlock, locking, mdl | ||
[8 Jul 2009 16:17]
Matthias Leich
Grammar of creation of objects
Attachment: bug46044_data.zz (application/octet-stream, text), 214 bytes.
[8 Jul 2009 16:18]
Matthias Leich
Grammar for stress testing
Attachment: bug46044_sql.yy (application/octet-stream, text), 396 bytes.
[8 Jul 2009 16:22]
Matthias Leich
Test output + backtrace after kill -11 at the end
Attachment: bug.prt.gz (application/x-gzip, text), 31.75 KiB.
[8 Jul 2009 18:58]
Matthias Leich
There is a missing ";" within the alternative grammar. Correct would be: query: # The HIGH_PRIORITY and FOR UPDATE seem to be necessary CREATE TABLE my_table AS SELECT HIGH_PRIORITY A . _field FROM _table AS A LEFT JOIN _table AS B USING (`pk`) LIMIT _digit FOR UPDATE; SELECT SLEEP(1); DROP TABLE my_table | LOCK TABLE _table READ , _table READ; SELECT SLEEP(1); UNLOCK TABLES; And with this grammar the deadlock disappears. But this does not change the fact that the outcome of the test with the first grammar is strange: 1. http://dev.mysql.com/doc/refman/6.0/en/select.html ... If you use FOR UPDATE with a storage engine that uses page or row locks, ... We are using MyISAM so this should be irrelevant = have no effect. But why disappears the deadlock if the "FOR UPDATE" is removed. 2. CREATE TABLE commits at start and end of executing the CREATE TABLE base operations like store metadata and fill the table (If AS SELECT ...). So it should UNLOCK tables already locked by the current session. Therefore I think this means the missing balance "We have a LOCK TABLE but no UNLOCK" should be no problem.
[14 Aug 2009 20:10]
Konstantin Osipov
The most apparent offender: # 16:48:51 Thread 41 (process 5514): # 16:48:51 #0 0x00007fbf1858bdd9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 # 16:48:51 #1 0x0000000000b30105 in safe_cond_wait (cond=0x12534c0, mp=0x1253400, file=0xd69b20 "mdl.cc", line=1324) at thr_mutex.c:423 # 16:48:51 #2 0x000000000091ba0e in MDL_context::wait_for_locks (this=0x27b20a0) at mdl.cc:1324 # 16:48:51 #3 0x0000000000757cfb in recover_from_failed_open_table_attempt (thd=0x27b1fc8, table=0x27f4660, action=OT_BACK_OFF_AND_RETRY) at sql_base.cc:3516 # 16:48:51 #4 0x0000000000759c75 in open_tables (thd=0x27b1fc8, start=0x448a25f0, counter=0x448a262c, flags=2) at sql_base.cc:3805 # 16:48:51 #5 0x000000000075a285 in open_normal_and_derived_tables (thd=0x27b1fc8, tables=0x27f4660, flags=2) at sql_base.cc:4276 # 16:48:51 #6 0x000000000088433e in fill_schema_show_cols_or_idxs (thd=0x27b1fc8, tables=0x27f5258, schema_table=0x1132160, open_tables_state_backup=0x448a2ab0) at sql_show.cc:2953 # 16:48:51 #7 0x00000000008953a2 in get_all_tables (thd=0x27b1fc8, tables=0x27f5258, cond=0x0) at sql_show.cc:3361 # 16:48:51 #8 0x0000000000880d4a in get_schema_tables_result (join=0x26a8ca0, executed_place=PROCESSED_BY_JOIN_EXEC) at sql_show.cc:6510 # 16:48:51 #9 0x00000000007a082e in JOIN::exec (this=0x26a8ca0) at sql_select.cc:2393 # 16:48:51 #10 0x000000000079d004 in mysql_select (thd=0x27b1fc8, rref_pointer_array=0x27b3f80, tables=0x27f5258, wild_num=0, fields=@0x27b3ea0, conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2686732800, # 16:48:51 result=0x28328c0, unit=0x27b3930, select_lex=0x27b3d98) at sql_select.cc:3070 # 16:48:51 #11 0x00000000007a2887 in handle_select (thd=0x27b1fc8, lex=0x27b3890, result=0x28328c0, setup_tables_done_option=0) at sql_select.cc:305 # 16:48:51 #12 0x00000000006fdb07 in execute_sqlcom_select (thd=0x27b1fc8, all_tables=0x27f5258) at sql_parse.cc:4976 # 16:48:51 #13 0x00000000006ff359 in mysql_execute_command (thd=0x27b1fc8) at sql_parse.cc:2161 # 16:48:51 #14 0x0000000000707914 in mysql_parse (thd=0x27b1fc8, inBuf=0x27f4070 "SHOW FIELDS FROM my_table", length=25, found_semicolon=0x448a4f30) at sql_parse.cc:5991 # 16:48:51 #15 0x00000000007084fe in dispatch_command (command=COM_QUERY, thd=0x27b1fc8, packet=0x27ceae9 "\n\t\t\tSHOW FIELDS FROM my_table", packet_length=29) at sql_parse.cc:1061 # 16:48:51 #16 0x00000000007099c2 in do_command (thd=0x27b1fc8) at sql_parse.cc:743 # 16:48:51 #17 0x00000000006f7038 in handle_one_connection (arg=0x27b1fc8) at sql_connect.cc:1158 # 16:48:51 #18 0x00007fbf18588040 in start_thread () from /lib64/libpthread.so.0 # 16:48:51 #19 0x00007fbf1752908d in clone () from /lib64/libc.so.6 # 16:48:51 #20 0x0000000000000000 in ?? () Matthias, are you sure the bug is not repeatable in 5.1?
[20 Aug 2009 11:40]
Dmitry Lenev
After investigating cause behind this problem I was able to come-up with the following simpler and IMO more natural test case for mysqltest tool which exposes the same issue: create table t1 (i int); connect (con1, localhost, root,,); connection default; lock tables t1 write; connection con1; --send create table t2 select * from t1; connection default; --sleep 2 --echo # The below statement hangs causing deadlock. show fields from t2; connection con1; --reap
[27 Aug 2009 6:23]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/81655 2802 Dmitry Lenev 2009-08-27 Fix for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY FOR UPDATE". Deadlock occured when during execution of query to I_S we tried to open a table or its .FRM in order to get information about it and had to wait because we have encountered exclusive metadata lock on this table held by a DDL operation from another connection which in its turn waited for some resource currently owned by connection executing this I_S query. For example, this might have happened if one under LOCK TABLES executed I_S query targeted to particular table (which was not among locked) and also concurrently tried to create this table using CREATE TABLE SELECT which had to wait for one of tables locked by the first connection. Another situation in which deadlock might have occured is when I_S query, which was executed as part of transaction, tried to get information about table which just has been dropped by concurrent DROP TABLES executed under LOCK TABLES and this DROP TABLES for its completion also had to wait transaction from the first connection. This problem stemmed from the fact that opening of tables/.FRMs for I_S filling is happening outside of connection's main MDL_context so code which tries to detect deadlocks due to conflicting metadata locks doesn't work in this case. Indeed, this led to deadlocks when during I_S filling we tried to wait for conflicting metadata lock to go away, while its owner was waiting for some resource held by connection executing I_S query. This patch solves this problem by avoiding waiting in such situation. Instead we skip this table and produce warning that information about it was omitted from I_S due to concurrent DDL operation. We still wait for conflicting metadata lock to go away when it is known that deadlock is not possible (i.e. when connection executing I_S query does not hold any metadata or table-level locks). Basically, we apply our standard deadlock avoidance technique for metadata locks to the process of filling of I_S tables but replace ER_LOCK_DEADLOCK error with a warning. Note that this change is supposed to be safe for 'mysqldump' since the only its mode which is affected by this change is --single-transaction mode is not safe in the presence of concurrent DDL anyway (and this fact is documented). Other modes are unaffected because they either use SHOW TABLES/SELECT * FROM I_S.TABLE_NAMES which do not take any metadata locks in the process of I_S table filling and thus cannot skip tables or execute I_S queries for tables which were previously locked by LOCK TABLES (or in the presence of global read lock) which excludes possibility of encountering conflicting metadata lock. @ mysql-test/r/mdl_sync.result Added test for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY FOR UPDATE". @ mysql-test/t/mdl_sync.test Added test for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY FOR UPDATE". @ sql/mysql_priv.h Added a new flag for open_table() call which allows it to fail with an error in cases when conflicting metadata lock is discovered instead of waiting until this lock goes away. @ sql/share/errmsg-utf8.txt Added error/warning message to be generated in cases when information about table is omitted from I_S since there is conflicting metadata lock on the table. @ sql/share/errmsg.txt Added error/warning message to be generated in cases when information about table is omitted from I_S since there is conflicting metadata lock on the table. @ sql/sql_base.cc Added a new flag for open_table() call which allows it to fail with an error in cases when conflicting metadata lock is discovered instead of waiting until this lock goes away. @ sql/sql_show.cc When we are opening a table (or just .FRM) in order to fill I_S with information about this table and encounter conflicting metadata lock waiting for this lock to go away can lead to a deadlock in some situations (under LOCK TABLES, within transaction, etc.). To avoid these deadlocks we detect such situations and don't do waiting. Instead, we skip table for which we have conflicting metadata lock, thus omitting information about it from I_S table, and produce an appropriate warning. Also tried to clarify error handling in fill_schema_table_from_frm().
[27 Aug 2009 6:25]
Dmitry Lenev
Fix for this bug was queued into mysql-next-bugfixing tree.
[28 Aug 2009 9:53]
Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090828095047-jbux7pie1yjnwpce) (version source revid:dlenev@mysql.com-20090827062217-rgx1i5eo3bcrd1v8) (merge vers: 5.4.4-alpha) (pib:11)
[29 Aug 2009 23:28]
Paul DuBois
Deadlock could occur for INFORMATION_SCHEMA queries when execution attempted to open a table or its .frm file.
[29 Aug 2009 23:29]
Paul DuBois
Noted in 5.4.4 changelog.
[9 Dec 2009 22:07]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/93402 3014 Konstantin Osipov 2009-12-09 Backport of: ------------------------------------------------------------ revno: 2617.68.7 committer: Dmitry Lenev <dlenev@mysql.com> branch nick: mysql-next-bg46044 timestamp: Thu 2009-08-27 10:22:17 +0400 message: Fix for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY FOR UPDATE". Deadlock occured when during execution of query to I_S we tried to open a table or its .FRM in order to get information about it and had to wait because we have encountered exclusive metadata lock on this table held by a DDL operation from another connection which in its turn waited for some resource currently owned by connection executing this I_S query. For example, this might have happened if one under LOCK TABLES executed I_S query targeted to particular table (which was not among locked) and also concurrently tried to create this table using CREATE TABLE SELECT which had to wait for one of tables locked by the first connection. Another situation in which deadlock might have occured is when I_S query, which was executed as part of transaction, tried to get information about table which just has been dropped by concurrent DROP TABLES executed under LOCK TABLES and this DROP TABLES for its completion also had to wait transaction from the first connection. This problem stemmed from the fact that opening of tables/.FRMs for I_S filling is happening outside of connection's main MDL_context so code which tries to detect deadlocks due to conflicting metadata locks doesn't work in this case. Indeed, this led to deadlocks when during I_S filling we tried to wait for conflicting metadata lock to go away, while its owner was waiting for some resource held by connection executing I_S query. This patch solves this problem by avoiding waiting in such situation. Instead we skip this table and produce warning that information about it was omitted from I_S due to concurrent DDL operation. We still wait for conflicting metadata lock to go away when it is known that deadlock is not possible (i.e. when connection executing I_S query does not hold any metadata or table-level locks). Basically, we apply our standard deadlock avoidance technique for metadata locks to the process of filling of I_S tables but replace ER_LOCK_DEADLOCK error with a warning. Note that this change is supposed to be safe for 'mysqldump' since the only its mode which is affected by this change is --single-transaction mode is not safe in the presence of concurrent DDL anyway (and this fact is documented). Other modes are unaffected because they either use SHOW TABLES/SELECT * FROM I_S.TABLE_NAMES which do not take any metadata locks in the process of I_S table filling and thus cannot skip tables or execute I_S queries for tables which were previously locked by LOCK TABLES (or in the presence of global read lock) which excludes possibility of encountering conflicting metadata lock. @ mysql-test/r/mdl_sync.result Added test for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY FOR UPDATE". @ mysql-test/t/mdl_sync.test Added test for bug #46044 "MDL deadlock on LOCK TABLE + CREATE TABLE HIGH_PRIORITY FOR UPDATE". @ sql/mysql_priv.h Added a new flag for open_table() call which allows it to fail with an error in cases when conflicting metadata lock is discovered instead of waiting until this lock goes away. @ sql/share/errmsg-utf8.txt Added error/warning message to be generated in cases when information about table is omitted from I_S since there is conflicting metadata lock on the table. @ sql/share/errmsg.txt Added error/warning message to be generated in cases when information about table is omitted from I_S since there is conflicting metadata lock on the table. @ sql/sql_base.cc Added a new flag for open_table() call which allows it to fail with an error in cases when conflicting metadata lock is discovered instead of waiting until this lock goes away. @ sql/sql_show.cc When we are opening a table (or just .FRM) in order to fill I_S with information about this table and encounter conflicting metadata lock waiting for this lock to go away can lead to a deadlock in some situations (under LOCK TABLES, within transaction, etc.). To avoid these deadlocks we detect such situations and don't do waiting. Instead, we skip table for which we have conflicting metadata lock, thus omitting information about it from I_S table, and produce an appropriate warning.
[16 Feb 2010 16:46]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100216101445-2ofzkh48aq2e0e8o) (version source revid:kostja@sun.com-20091211154405-c9yhiewr9o5d20rq) (merge vers: 6.0.14-alpha) (pib:16)
[16 Feb 2010 16:55]
Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100216101208-33qkfwdr0tep3pf2) (version source revid:kostja@sun.com-20091209154842-k3dh01k9uj8sasz9) (pib:16)
[6 Mar 2010 11:06]
Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:vvaintroub@mysql.com-20100216221947-luyhph0txl2c5tc8) (merge vers: 5.5.99-m3) (pib:16)
[7 Mar 2010 2:08]
Paul DuBois
Noted in 5.5.3, 6.0.14 changelogs.
[13 Apr 2010 5:03]
Paul DuBois
Correction: Not present in any 5.5.x release. 5.5.3 changelog entry removed.
Description: I get a deadlock (= all "worker" sessions wait for a table or a lock) after a short runtime in RQG. The CPU consumption is ~ 0%. Grammar for object creation: $tables = { rows => [ 1, 10 ] }; $fields = { types => [ 'int' ], indexes => [undef ] }; $data = { numbers => [ 'null' ], temporals => [ undef ], strings => [ 'letter' ] } Grammar for concurrent SQL: query: # The HIGH_PRIORITY and FOR UPDATE seem to be necessary. # Just the SELECT .... without the CREATE TABLE etc. is not sufficient. CREATE TABLE my_table AS SELECT HIGH_PRIORITY A . _field FROM _table AS A LEFT JOIN _table AS B USING (`pk`) LIMIT _digit FOR UPDATE; SELECT SLEEP(1); DROP TABLE my_table | LOCK TABLE _table READ , _table READ; # Locking two tables seems to be necessary gendata.pl contains direct after the creation of a table following additional statements which generate a VIEW per every base table: my $create_result = output ("CREATE VIEW `v$table->[TABLE_NAME]` AS SELECT * FROM `$table->[TABLE_NAME]`"); if ($create_result > 1) { say("# Unable to create view $table->[TABLE_NAME], skipping..."); next; } My command: ---------- perl runall.pl --mem --basedir=/work2/6.0/azalea/ \ --threads=100 --queries=1000 --debug \ --mysqld=--table-lock-wait-timeout=5 \ --mysqld=--innodb-lock-wait-timeout=5 \ --gendata=conf/ml_bugnnnnn_data.zz \ --grammar=conf/ml_bugnnnnn_sql.yy --seed=0 State after 1 - 2 minutes: -------------------------- The growth of general_log.CSV has ended. SHOW OPEN TABLES FROM test ; Database Table In_use Name_locked test table1_int_autoinc 20 0 test vtable10_int_autoinc 0 0 test vtable1_int_autoinc 0 0 test table10_int_autoinc 20 0 PROCESSLIST: 19 session in state "Table lock" 81 sessions in state "Waiting for table" Most probably important/suspicious things: 1. Only MyISAM and no transactional engines or partitioning are involved. The VIEWs seem to be needed 2. IMHO a CREATE TABLE ... AS SELECT with "HIGH_PRIORITY" and "FOR UPDATE" looks somehow very unusual. I can at least imagine that HIGH_PRIORITY could give sometimes benefits. But I am unsure if it is really intended that "FOR UPDATE" in such a CREATE TABLE is allowed. Whenever I tried to remove the "HIGH_PRIORITY" or the "FOR UPDATE" the deadlock disappeared. At looks as if the "--table-lock-wait-timeout=5" has no impact on the outcome of the test at all. This is a pity because it could at least in theory break the deadlock. I will attach a file with backtraces which I got after kill -11 <server process>. My environment: - MySQL azalea last change 2009-07-02 - BUILD/compile-pentium64-debug-max - Linux OpenSuSE 11.0 (64 Bit) - Intel Core2Duo In case you judge that the current grammar is expected to create deadlocks because of missing UNLOCK tables. Please use the following grammar query: # The HIGH_PRIORITY and FOR UPDATE seem to be necessary CREATE TABLE my_table AS SELECT HIGH_PRIORITY A . _field FROM _table AS A LEFT JOIN _table AS B USING (`pk`) LIMIT _digit FOR UPDATE; SELECT SLEEP(1); DROP TABLE my_table | LOCK TABLE _table READ , _table READ; SELECT SLEEP(1) UNLOCK TABLES; and observe the deadlock. How to repeat: Please use the attached grammar files. Suggested fix: Please modify the server so that "--table-lock-wait-timeout=5" resolves this deadlock. I am aware that especially the CREATE TABLE statement looks ugly and it is rather rare that somebody else runs similar stuff.