Bug #30977 | Concurrent statement using stored function and DROP FUNCTION breaks SBR | ||
---|---|---|---|
Submitted: | 12 Sep 2007 15:03 | Modified: | 14 Dec 2010 19:12 |
Reporter: | Dmitry Lenev | Email Updates: | |
Status: | Closed | Impact on me: | |
Category: | MySQL Server: Locking | Severity: | S3 (Non-critical) |
Version: | 5.0.50-bk, 5.1.23-bk | OS: | Any |
Assigned to: | Jon Olav Hauglid | CPU Architecture: | Any |
Tags: | mdl |
[12 Sep 2007 15:03]
Dmitry Lenev
[17 Sep 2007 13:05]
MySQL Verification Team
Thank you for the bug report.
[26 Sep 2007 9:53]
Konstantin Osipov
Will be fixed in 5.2 only
[10 Jun 2009 11:07]
Magne Mæhre
Reassigning. Jon Olav and I are looking at it..
[16 Jun 2009 9:50]
Konstantin Osipov
A code review should cover the following points in the implementation: 1) CREATE PROCEDURE/CREATE FUNCTION/DROP PROCEDURE/DROP FUNCTION, no LOCK TABLES. Verify that the implementation takes an exclusive MDL on routine name. It must be taken only after committing the pending transaction, to unlock all other MDL locks taken by this connection, if any. lock_routine_name() + open_proc_table_for_update() should happen in the same try-and-back-off attempt. Make sure that if there is a back off during open_proc_table_for_update(), it doesn't remove the MDL request for the routine name from the context. In other words, make sure we always have an exclusive lock on the name of the routine and a shared lock on mysql.proc in these statements. 2) CREATE PROCEDURE/FUNCTION etc. under LOCK TABLES -> not allowed Right now it is allowed, but mysql.proc must also be locked with LOCK TABLES. Should be prohibited. If we try to allow it, we can easily have deadlocks. 3) SELECT and other DML (no LOCK TABLES): - adds routine names to the prelocked list * performs addition and takes the MDL locks *before* opening the function (that is, before reading its list of prelocked tables, etc.). - if encounters a conflicting exclusive lock when trying to take a shared lock on the routine name, behaves the same way as for tables: - if inside a multi-statement transaction with other MDL locks in MDL context, ER_LOCK_DEADLOCK - if the first statement of a transaction, or auto-commit mode, back off and retry. - locks should be taken on on all routines used, directly or indirectly: - directly in the statement SQL - via a view - via a trigger - via other stored routine 4) LOCK TABLES. No change. We don't take metadata locks on routine names when entering LOCK TABLES, there is no syntax to specify names explicitly, and no way for us to infer what routines will be used down the road. 5) SELECT and other DML under LOCK TABLES. This mode has the same behavior as if it was a subsequent statement inside a transaction: ER_LOCK_DEADLOCK on lock conflict. But we must make sure that we do take shared metadata locks on routines. Verify that there is no memory hog. 6) Verify sufficient test coverage for 1-2-3-4-5 above. 7) Verify that changeset comments highlight all incompatible changes, such as all new scenarios when ER_LOCK_DEADLOCK can occur, and all scenarios that are no longer allowed.
[7 Jul 2009 15:01]
Konstantin Osipov
Bug#40419 was marked a duplicate of this bug.
[22 Aug 2009 9:04]
Konstantin Osipov
Bug#12815 is a duplicate of this bug.
[25 Aug 2009 7:23]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/81475 2876 Jon Olav Hauglid 2009-08-25 A pre-requisite patch for Bug#30977 "Concurrent statement using stored function and DROP FUNCTION breaks SBR". This patch changes the MDL API by introducing a namespace for lock keys: MDL_TABLE for tables and views and MDL_PROCEDURE for stored procedures and functions. The latter is needed for the fix for Bug#30977.
[28 Aug 2009 9:54]
Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090828095047-jbux7pie1yjnwpce) (version source revid:mats@sun.com-20090826091405-64t3gq3mxbbhzwt8) (merge vers: 5.4.4-alpha) (pib:11)
[2 Sep 2009 9:42]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/82188 2804 Dmitry Lenev 2009-09-02 Pre-requisite patch for fixing bug #30977 "Concurrent statement using stored function and DROP FUNCTION breaks SBR". Changed open_tables() implementation to process stored routines only after tables which are directly used by statement were processed. Thanks to this change we can now start taking shared metadata locks on stored routines used in statement without breaking one of metadata locking invariants for statements like "CREATE TABLE ... SELECT f1()". The problem was that for such statements we take exclusive metadata lock on table being created and this invariant states that such locks should not be taken if connection holds any other locks. So processing "f1()", which would have included taking shared metadata lock on this routine, before processing table list element for table being created would have led to breaking this invariant. @ sql/sql_base.cc Refactored open_tables() implementation to process stored routines only after tables which are directly used by statement were processed. To achieve this moved handling of routines in open_tables() (done by open_routines() calls) to one place. This involved moving calls to open_routines() out of loop which iterates over tables to a new separate loop. And in its turn this allowed to split handling of tables and views to an auxiliary function, which made code in open_tables() simplier and more easy to understand. @ storage/myisammrg/ha_myisammrg.cc Ensure that when we are adding or excluding elements for child tables to/from the statement's table list LEX::query_tables_last is updated to correct value so more elements can be added to the table list.
[3 Sep 2009 9:45]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/82307 2805 Dmitry Lenev 2009-09-03 Pre-requisite patch for fixing bug #30977 "Concurrent statement using stored function and DROP FUNCTION breaks SBR". Added MDL_request for stored routine as member to Sroutine_hash_entry in order to be able perform metadata locking for stored routines in future (Sroutine_hash_entry is an equivalent of TABLE_LIST class for stored routines). QQ: May be it makes sense to try using MDL_key as key for functions responsible for SP loading and caching? By doing this we might got rid of necessity to construct sp_name::m_qname at least in some cases... @ sql/mdl.cc Introduced version of MDL_request::init() method which initializes lock request using pre-built MDL key. @ sql/mdl.h Extended enum_mdl_namespace enum with values which correspond to namespaces for stored functions and triggers. Added MDL_key::mdl_namespace() getter. Also added version of MDL_request::init() method which initializes lock request using pre-built MDL key. @ sql/sp.cc Added MDL_request for stored routine as member to Sroutine_hash_entry. Changed code to use MDL_key from this request as a key for LEX::sroutines set. Removed separate "key" member from Sroutine_hash_entry as it became unnecessary. @ sql/sp.h Added MDL_request for stored routine as member to Sroutine_hash_entry in order to be able perform metadata locking for stored routines in future (Sroutine_hash_entry is an equivalent of TABLE_LIST class for stored routines). Removed Sroutine_hash_entry::key member as now we can use MDL_key from this request as a key for LEX::sroutines set. @ sql/sp_head.cc Removed sp_name::m_sroutines_key member and set_routine_type() method. Since key for routine in LEX::sroutines set has no longer sp_name::m_qname as suffix we won't save anything by creating it at sp_name construction time. Adjusted sp_name constructor used for creating temporary objects for lookups in SP-cache to accept MDL_key as parameter and to avoid any memory allocation. Finally, removed sp_head::m_soutines_key member for reasons similar to why sp_name::m_sroutines_key was removed. @ sql/sp_head.h Removed sp_name::m_sroutines_key member and set_routine_type() method. Since key for routine in LEX::sroutines set has no longer sp_name::m_qname as suffix we won't save anything by creating it at sp_name construction time. Adjusted sp_name constructor used for creating temporary objects for lookups in SP-cache to accept MDL_key as parameter and to avoid any memory allocation. Finally, removed sp_head::m_soutines_key member for reasons similar to why sp_name::m_sroutines_key was removed. @ sql/sql_base.cc Adjusted code to the fact that we now use MDL_key from Sroutine_hash_entry::mdl_request as a key for LEX::sroutines set. @ sql/sql_trigger.cc sp_add_used_routine() now takes MDL_key as parameter as now we use instance of this class as a key for LEX::sroutines set.
[16 Sep 2009 5:35]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/83399 2818 Dmitry Lenev 2009-09-16 Pre-requisite patch for fixing bug #30977 "Concurrent statement using stored function and DROP FUNCTION breaks SBR". CREATE TABLE SELECT statements take exclusive metadata lock on table being created. Invariant of metadata locking subsystem states that such lock should be taken before taking any kind of shared locks. Once metadata locks on stored routines are introduced statements like "CREATE TABLE ... SELECT f1()" will break this invariant by taking shared locks on routines before exclusive lock on target table. To avoid this, open_tables() is reworked to process tables which are directly used by the statement before stored routines are processed. @ sql/sql_base.cc Refactored open_tables() implementation to process stored routines only after tables which are directly used by statement were processed. To achieve this moved handling of routines in open_tables() out of loop which iterates over tables to a new separate loop. And in its turn this allowed to split handling of particular table or view to an auxiliary function, which made code in open_tables() simpler and more easy to understand. @ storage/myisammrg/ha_myisammrg.cc Ensure that when we are adding or excluding elements for child tables to/from the statement's table list LEX::query_tables_last is updated to correct value so more elements can be added to the table list.
[16 Sep 2009 13:26]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/83485 2819 Dmitry Lenev 2009-09-16 Pre-requisite patch for fixing bug #30977 "Concurrent statement using stored function and DROP FUNCTION breaks SBR". Added MDL_request for stored routine as member to Sroutine_hash_entry in order to be able perform metadata locking for stored routines in future (Sroutine_hash_entry is an equivalent of TABLE_LIST class for stored routines). @ sql/mdl.cc Introduced version of MDL_request::init() method which initializes lock request using pre-built MDL key. MDL_key::table_name/table_name_length() getters were renamed to reflect the fact that MDL_key objects are now created not only for tables. @ sql/mdl.h Extended enum_mdl_namespace enum with values which correspond to namespaces for stored functions and triggers. Renamed MDL_key::table_name/table_name_length() getters to MDL_key::name() and name_length() correspondingly to reflect the fact that MDL_key objects are now created not only for tables. Added MDL_key::mdl_namespace() getter. Also added version of MDL_request::init() method which initializes lock request using pre-built MDL key. @ sql/si_objects.cc MDL_key::table_name/table_name_length() getters were renamed to reflect the fact that MDL_key objects are now created not only for tables. @ sql/sp.cc Added MDL_request for stored routine as member to Sroutine_hash_entry. Changed code to use MDL_key from this request as a key for LEX::sroutines set. Removed separate "key" member from Sroutine_hash_entry as it became unnecessary. @ sql/sp.h Added MDL_request for stored routine as member to Sroutine_hash_entry in order to be able perform metadata locking for stored routines in future (Sroutine_hash_entry is an equivalent of TABLE_LIST class for stored routines). Removed Sroutine_hash_entry::key member as now we can use MDL_key from this request as a key for LEX::sroutines set. @ sql/sp_head.cc Removed sp_name::m_sroutines_key member and set_routine_type() method. Since key for routine in LEX::sroutines set has no longer sp_name::m_qname as suffix we won't save anything by creating it at sp_name construction time. Adjusted sp_name constructor used for creating temporary objects for lookups in SP-cache to accept MDL_key as parameter and to avoid any memory allocation. Finally, removed sp_head::m_soutines_key member for reasons similar to why sp_name::m_sroutines_key was removed. @ sql/sp_head.h Removed sp_name::m_sroutines_key member and set_routine_type() method. Since key for routine in LEX::sroutines set has no longer sp_name::m_qname as suffix we won't save anything by creating it at sp_name construction time. Adjusted sp_name constructor used for creating temporary objects for lookups in SP-cache to accept MDL_key as parameter and to avoid any memory allocation. Finally, removed sp_head::m_soutines_key member for reasons similar to why sp_name::m_sroutines_key was removed. @ sql/sql_base.cc Adjusted code to the fact that we now use MDL_key from Sroutine_hash_entry::mdl_request as a key for LEX::sroutines set. MDL_key::table_name/table_name_length() getters were renamed to reflect the fact that MDL_key objects are now created not only for tables. @ sql/sql_trigger.cc sp_add_used_routine() now takes MDL_key as parameter as now we use instance of this class as a key for LEX::sroutines set.
[16 Sep 2009 14:27]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/83497 2820 Dmitry Lenev 2009-09-16 Follow-up for one of pre-requisite patches for fixing bug #30977 "Concurrent statement using stored function and DROP FUNCTION breaks SBR". Made enum_mdl_namespace enum part of MDL_key class and removed MDL_ prefix from the names of enum members. In order to do the latter changed name of PROCEDURE symbol to PROCEDURE_SYM (otherwise macro which was automatically generated for this symbol conflicted with MDL_key::PROCEDURE enum member).
[17 Sep 2009 3:21]
Dmitry Lenev
The above pre-requisite patches were queued into mysql-6.0-codebase-bugfixing tree.
[17 Sep 2009 13:58]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/83613
[30 Sep 2009 8:16]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20090929093622-1mooerbh12e97zux) (version source revid:alik@sun.com-20090923103200-kyo2bakdo6tfb2fb) (merge vers: 6.0.14-alpha) (pib:11)
[19 Oct 2009 14:05]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/87344
[9 Dec 2009 8:57]
Jon Olav Hauglid
Pushed prerequisite patch originally committed as http://lists.mysql.com/commits/81475 to mysql-next-4284 (5.6.0-beta).
[9 Dec 2009 22:05]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/93401 3017 Konstantin Osipov 2009-12-09 Backport of: ------------------------------------------------------------ revno: 2617.68.24 committer: Dmitry Lenev <dlenev@mysql.com> branch nick: mysql-next-bg-pre2-2 timestamp: Wed 2009-09-16 17:25:29 +0400 message: Pre-requisite patch for fixing bug #30977 "Concurrent statement using stored function and DROP FUNCTION breaks SBR". Added MDL_request for stored routine as member to Sroutine_hash_entry in order to be able perform metadata locking for stored routines in future (Sroutine_hash_entry is an equivalent of TABLE_LIST class for stored routines). (WL#4284, follow up fixes). @ sql/mdl.cc Introduced version of MDL_request::init() method which initializes lock request using pre-built MDL key. MDL_key::table_name/table_name_length() getters were renamed to reflect the fact that MDL_key objects are now created not only for tables. @ sql/mdl.h Extended enum_mdl_namespace enum with values which correspond to namespaces for stored functions and triggers. Renamed MDL_key::table_name/table_name_length() getters to MDL_key::name() and name_length() correspondingly to reflect the fact that MDL_key objects are now created not only for tables. Added MDL_key::mdl_namespace() getter. Also added version of MDL_request::init() method which initializes lock request using pre-built MDL key. @ sql/sp.cc Added MDL_request for stored routine as member to Sroutine_hash_entry. Changed code to use MDL_key from this request as a key for LEX::sroutines set. Removed separate "key" member from Sroutine_hash_entry as it became unnecessary. @ sql/sp.h Added MDL_request for stored routine as member to Sroutine_hash_entry in order to be able perform metadata locking for stored routines in future (Sroutine_hash_entry is an equivalent of TABLE_LIST class for stored routines). Removed Sroutine_hash_entry::key member as now we can use MDL_key from this request as a key for LEX::sroutines set. @ sql/sp_head.cc Removed sp_name::m_sroutines_key member and set_routine_type() method. Since key for routine in LEX::sroutines set has no longer sp_name::m_qname as suffix we won't save anything by creating it at sp_name construction time. Adjusted sp_name constructor used for creating temporary objects for lookups in SP-cache to accept MDL_key as parameter and to avoid any memory allocation. Finally, removed sp_head::m_soutines_key member for reasons similar to why sp_name::m_sroutines_key was removed @ sql/sp_head.h Removed sp_name::m_sroutines_key member and set_routine_type() method. Since key for routine in LEX::sroutines set has no longer sp_name::m_qname as suffix we won't save anything by creating it at sp_name construction time. Adjusted sp_name constructor used for creating temporary objects for lookups in SP-cache to accept MDL_key as parameter and to avoid any memory allocation. Finally, removed sp_head::m_soutines_key member for reasons similar to why sp_name::m_sroutines_key was removed. @ sql/sql_base.cc Adjusted code to the fact that we now use MDL_key from Sroutine_hash_entry::mdl_request as a key for LEX::sroutines set. MDL_key::table_name/table_name_length() getters were renamed to reflect the fact that MDL_key objects are now created not only for tables. @ sql/sql_trigger.cc sp_add_used_routine() now takes MDL_key as parameter as now we use instance of this class as a key for LEX::sroutines set.
[9 Dec 2009 22:08]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/93403 3016 Konstantin Osipov 2009-12-09 ------------------------------------------------------------ revno: 2617.68.23 committer: Dmitry Lenev <dlenev@mysql.com> branch nick: mysql-next-bg-pre1 timestamp: Wed 2009-09-16 09:34:42 +0400 message: Pre-requisite patch for fixing bug #30977 "Concurrent statement using stored function and DROP FUNCTION breaks SBR". CREATE TABLE SELECT statements take exclusive metadata lock on table being created. Invariant of metadata locking subsystem states that such lock should be taken before taking any kind of shared locks. Once metadata locks on stored routines are introduced statements like "CREATE TABLE ... SELECT f1()" will break this invariant by taking shared locks on routines before exclusive lock on target table. To avoid this, open_tables() is reworked to process tables which are directly used by the statement before stored routines are processed. @ sql/sql_base.cc Refactored open_tables() implementation to process stored routines only after tables which are directly used by statement were processed. To achieve this moved handling of routines in open_tables() out of loop which iterates over tables to a new separate loop. And in its turn this allowed to split handling of particular table or view to an auxiliary function, which made code in open_tables() simpler and more easy to understand.
[10 Dec 2009 8:22]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/93445 3018 Konstantin Osipov 2009-12-10 Backport of: ------------------------------------------------------------ revno: 2617.68.25 committer: Dmitry Lenev <dlenev@mysql.com> branch nick: mysql-next-bg-pre2-2 timestamp: Wed 2009-09-16 18:26:50 +0400 message: Follow-up for one of pre-requisite patches for fixing bug #30977 "Concurrent statement using stored function and DROP FUNCTION breaks SBR". Made enum_mdl_namespace enum part of MDL_key class and removed MDL_ prefix from the names of enum members. In order to do the latter changed name of PROCEDURE symbol to PROCEDURE_SYM (otherwise macro which was automatically generated for this symbol conflicted with MDL_key::PROCEDURE enum member).
[29 Dec 2009 12:22]
Konstantin Osipov
Pushed in next-4284.
[30 Dec 2009 20:23]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/95823 3745 Konstantin Osipov 2009-12-30 [merge] Merge the fix for Bug#30977 from next-4284.
[26 Jan 2010 10:21]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/98168 3055 Jon Olav Hauglid 2010-01-26 Additional test coverage for Bug#30977 Concurrent statement using stored function and DROP FUNCTION breaks SBR Bug#48246 assert in close_thread_table
[16 Feb 2010 16:46]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100216101445-2ofzkh48aq2e0e8o) (version source revid:jon.hauglid@sun.com-20100120100711-9vpxnuwkwgh55e9v) (merge vers: 6.0.14-alpha) (pib:16)
[16 Feb 2010 16:55]
Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100216101208-33qkfwdr0tep3pf2) (version source revid:jon.hauglid@sun.com-20100120081731-1itayyo38bxgoi9j) (pib:16)
[2 Mar 2010 1:12]
Paul DuBois
Noted in 6.0.14 changelog. Concurrent statements using a stored function and DROP FUNCTION for that function could break statement-based replication. Setting report to Need Merge pending push of Celosia to release tree.
[6 Mar 2010 11:01]
Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:vvaintroub@mysql.com-20100216221947-luyhph0txl2c5tc8) (merge vers: 5.5.99-m3) (pib:16)
[6 Mar 2010 20:15]
Paul DuBois
Noted in 5.5.3 changelog.
[19 Apr 2010 13:35]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/106003 3004 Jon Olav Hauglid 2010-04-19 Additional test coverage for Bug#30977 Concurrent statement using stored function and DROP FUNCTION breaks SBR Bug#48246 assert in close_thread_table
[19 Apr 2010 13:37]
Jon Olav Hauglid
Patch containing extra test coverage pushed to mysql-trunk-runtime (Ver 5.5.4-m3). Since the patch contains no code changes, I'm leaving the bug as Closed.
[27 Apr 2010 9:45]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100427094135-5s49ecp3ckson6e2) (version source revid:alik@sun.com-20100427093843-uekr85qkd7orx12t) (merge vers: 6.0.14-alpha) (pib:16)
[27 Apr 2010 9:48]
Bugs System
Pushed into 5.5.5-m3 (revid:alik@sun.com-20100427093804-a2k3rrjpwu5jegu8) (version source revid:alik@sun.com-20100427093804-a2k3rrjpwu5jegu8) (merge vers: 5.5.5-m3) (pib:16)
[27 Apr 2010 9:50]
Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100427094036-38frbg3famdlvjup) (version source revid:alik@sun.com-20100427093825-92wc8b22d4yg34ju) (pib:16)
[5 Dec 2010 12:41]
Bugs System
Pushed into mysql-trunk 5.6.1 (revid:alexander.nozdrin@oracle.com-20101205122447-6x94l4fmslpbttxj) (version source revid:alexander.nozdrin@oracle.com-20101205122447-6x94l4fmslpbttxj) (merge vers: 5.6.1) (pib:23)
[14 Dec 2010 17:58]
Jon Stephens
User-visible effect is on replication/binary logging, so I'll take this one.
[14 Dec 2010 19:12]
Jon Stephens
Disregard previous comment -- fix has already been documented in 5.5.3/6.0.14 changelogs, no 5.6 changelog entry necessary. Closed.
[16 Dec 2010 22:30]
Bugs System
Pushed into mysql-5.5 5.5.9 (revid:jonathan.perkin@oracle.com-20101216101358-fyzr1epq95a3yett) (version source revid:jonathan.perkin@oracle.com-20101216101358-fyzr1epq95a3yett) (merge vers: 5.5.9) (pib:24)
[22 Dec 2010 22:28]
Paul DuBois
Addition to changelog entry (Per Jon Olav): DDL statements for stored routines are now prohibited while a LOCK TABLES statement is in effect.