Bug #51136 Crash in pthread_rwlock_rdlock on TEMPORARY + HANDLER + LOCK + SP
Submitted: 12 Feb 2010 9:21 Modified: 7 Mar 2010 0:58
Reporter: Philip Stoev Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Locking Severity:S2 (Serious)
Version:next-mr-4284 OS:Any
Assigned to: Dmitry Lenev CPU Architecture:Any

[12 Feb 2010 9:21] Philip Stoev
Description:
The following sequence of queries causes mysqld to crash as follows:

#2  0x0000000000642639 in handle_segfault (sig=11) at mysqld.cc:2727
#3  <signal handler called>
#4  0x000000315b00a72a in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
#5  0x000000000086c74e in MDL_lock::has_pending_conflicting_lock (this=0x40, type=MDL_SHARED) at mdl.cc:1092
#6  0x000000000086c7c8 in MDL_ticket::has_pending_conflicting_lock (this=0x2f432e0) at mdl.cc:2075
#7  0x00000000005525a5 in mysql_ha_flush (thd=0x2eee7c8) at sql_handler.cc:803
#8  0x00000000006b07b4 in open_tables (thd=0x2eee7c8, start=0x7f6268258c60, counter=0x7f6268258c9c, flags=0, prelocking_strategy=0x7f6268258ce0)
    at sql_base.cc:4467
#9  0x00000000006b0e49 in open_and_lock_tables_derived (thd=0x2eee7c8, tables=0x2f7ce28, derived=true, flags=0, prelocking_strategy=0x7f6268258ce0)
    at sql_base.cc:5124
#10 0x000000000065f9aa in open_and_lock_tables_derived (thd=0x2eee7c8, tables=0x2f7ce28, derived=true, flags=0) at ../../sql/mysql_priv.h:1573
#11 0x000000000065f9e5 in open_and_lock_tables (thd=0x2eee7c8, tables=0x2f7ce28) at mysql_priv.h:1583
#12 0x00000000006fa887 in mysql_insert (thd=0x2eee7c8, table_list=0x2f7ce28, fields=@0x2ef0bb0, values_list=@0x2ef0bf8, update_fields=@0x2ef0be0,
    update_values=@0x2ef0bc8, duplic=DUP_REPLACE, ignore=false) at sql_insert.cc:649
#13 0x00000000006577a3 in mysql_execute_command (thd=0x2eee7c8) at sql_parse.cc:2979
#14 0x000000000065c862 in mysql_parse (thd=0x2eee7c8, inBuf=0x2f7cd28 "REPLACE  INTO view1  ( `pk` ) VALUES ( 5 )", length=42,
    found_semicolon=0x7f626825ae90) at sql_parse.cc:5581
#15 0x000000000065d47b in dispatch_command (command=COM_QUERY, thd=0x2eee7c8, packet=0x2f74c99 "REPLACE  INTO view1  ( `pk` ) VALUES ( 5 )",
    packet_length=42) at sql_parse.cc:1023
#16 0x000000000065e91b in do_command (thd=0x2eee7c8) at sql_parse.cc:709
#17 0x000000000064cafb in do_handle_one_connection (thd_arg=0x2eee7c8) at sql_connect.cc:1174
#18 0x000000000064cbca in handle_one_connection (arg=0x2eee7c8) at sql_connect.cc:1113
#19 0x00000000009db106 in pfs_spawn_thread (arg=0x2f3afa8) at pfs.cc:1011
#20 0x000000315b0073da in start_thread () from /lib64/libpthread.so.0
#21 0x000000315a4e627d in clone () from /lib64/libc.so.6

How to repeat:
--disable_abort_on_error
CREATE TEMPORARY TABLE IF NOT EXISTS temp1 (f1 integer);
CREATE TABLE IF NOT EXISTS t1 (f1 integer);
HANDLER t1  OPEN;
LOCK TABLES temp1 READ;
DELIMITER |;
CREATE PROCEDURE p1  () BEGIN SELECT 1 ; END |
DELIMITER ;|
REPLACE  INTO view1  ( `pk` ) VALUES ( 5 );
[12 Feb 2010 9:28] Philip Stoev
Here is a very similar test case

--disable_abort_on_error
CREATE TABLE `table10_int_autoinc` ( `col_int_key` int, pk integer auto_increment, `col_int` int, /*Indices*/ key (`col_int_key` ), primary key (pk));
CREATE DATABASE IF NOT EXISTS testdb_N  COLLATE = utf8_bin;
CREATE TEMPORARY TABLE IF NOT EXISTS testdb_N . t1_temp1_N  LIKE test.table10_int_autoinc;
CREATE TABLE IF NOT EXISTS testdb_N . t1_base1_N  LIKE test.table10_int_autoinc;
HANDLER testdb_N . t1_base1_N  OPEN;
LOCK TABLES testdb_N . t1_temp1_N  AS o READ;
DELIMITER |;
CREATE PROCEDURE testdb_N . p1_1_N  () BEGIN SELECT 1 ; END|
REPLACE  INTO testdb_N . t1_view2_N  ( `pk` ) VALUES ( 5 );

That crashes as follows:

#4  0x00000000006b1d62 in MDL_key::name (this=0x48) at mdl.h:185
#5  0x000000000086d4ff in MDL_context::release_lock (this=0x18fe898, ticket=0x1963e20) at mdl.cc:1915
#6  0x00000000005523ae in mysql_ha_close_table (thd=0x18fe7c8, tables=0x1996158) at sql_handler.cc:135
#7  0x0000000000552487 in mysql_ha_cleanup (thd=0x18fe7c8) at sql_handler.cc:832
#8  0x0000000000629a52 in THD::cleanup (this=0x18fe7c8) at sql_class.cc:996
#9  0x0000000000642849 in unlink_thd (thd=0x18fe7c8) at mysqld.cc:2036
#10 0x00000000006428fd in one_thread_per_connection_end (thd=0x18fe7c8, put_in_cache=true) at mysqld.cc:2123
#11 0x000000000064cb71 in do_handle_one_connection (thd_arg=0x18fe7c8) at sql_connect.cc:1181
#12 0x000000000064cbca in handle_one_connection (arg=0x18fe7c8) at sql_connect.cc:1113
#13 0x00000000009db106 in pfs_spawn_thread (arg=0x194afa8) at pfs.cc:1011
#14 0x000000315b0073da in start_thread () from /lib64/libpthread.so.0
#15 0x000000315a4e627d in clone () from /lib64/libc.so.6
[12 Feb 2010 9:47] Philip Stoev
Here is yet another one , with FLUSH instead of a stored procedure:

--disable_abort_on_error
CREATE TABLE `table1_int_autoinc` ( `col_int` int, `col_int_key` int, pk integer auto_increment, /*Indices*/ key (`col_int_key` ), primary key (pk));
CREATE TABLE `table10_int_autoinc` ( `col_int_key` int, pk integer auto_increment, `col_int` int, /*Indices*/ key (`col_int_key` ), primary key (pk));
CREATE SCHEMA  testdb_N  CHARACTER SET = utf8;
CREATE TEMPORARY TABLE IF NOT EXISTS testdb_N . t1_temp1_N  LIKE test.table1_int_autoinc;
CREATE TABLE IF NOT EXISTS testdb_N . t1_base2_N  LIKE test.table10_int_autoinc;
HANDLER testdb_N . t1_base2_N  OPEN AS A;
LOCK TABLES testdb_N . t1_temp1_N  AS t READ;
FLUSH TABLE testdb_S . t1_part2_S;
SELECT 1 FROM testdb_N . t1_view2_N  WHERE SLEEP( 0.5 * 0.47 * 0 ) = 0 LIMIT 1;
[14 Feb 2010 17:50] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/100307

3096 Dmitry Lenev	2010-02-14
      Fix for bug #51136 "Crash in pthread_rwlock_rdlock on 
      TEMPORARY + HANDLER + LOCK + SP".
      
      Server crashed when one: 
      1) Opened HANDLER or acquired global read lock
      2) Then locked one or several temporary tables with
         LOCK TABLES statement (but no base tables).
      3) Then issued any statement causing commit (explicit 
         or implicit).
      4) Issued statement which should have closed HANDLER
         or released global read lock.
         
      The problem was that when entering LOCK TABLES mode in the
      scenario described above we incorrectly set transactional
      MDL sentinel to zero. As result during commit all metadata 
      locks were released (including lock for open HANDLER or
      global metadata shared lock). Indeed, attempt to release
      metadata lock for the second time which happened during
      HANLDER CLOSE or during release of GLR caused crash.
      
      This patch fixes problem by changing MDL_context's
      set_trans_sentinel() method to set sentinel to correct 
      value (it should point to the most recent ticket).
      
      Question for reviewer is marked by QQ.
     @ mysql-test/include/handler.inc
        Added test for bug #51136 "Crash in pthread_rwlock_rdlock on 
        TEMPORARY + HANDLER + LOCK + SP".
     @ mysql-test/r/handler_innodb.result
        Updated test results (see include/handler.inc).
     @ mysql-test/r/handler_myisam.result
        Updated test results (see include/handler.inc).
     @ sql/mdl.h
        When setting new value of transactional sentinel use 
        pointer to the most recent ticket instead of value 
        returned by MDL_context::mdl_savepoint(). 
        This allows to handle correctly situation when the new 
        value of sentinel should be the same as its current value 
        (MDL_context::mdl_savepoint() returns NULL in this case).
[15 Feb 2010 11:25] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/100345

3097 Dmitry Lenev	2010-02-15
      Fix for bug #51136 "Crash in pthread_rwlock_rdlock on 
      TEMPORARY + HANDLER + LOCK + SP".
      
      Server crashed when one: 
      1) Opened HANDLER or acquired global read lock
      2) Then locked one or several temporary tables with
         LOCK TABLES statement (but no base tables).
      3) Then issued any statement causing commit (explicit 
         or implicit).
      4) Issued statement which should have closed HANDLER
         or released global read lock.
         
      The problem was that when entering LOCK TABLES mode in the
      scenario described above we incorrectly set transactional
      MDL sentinel to zero. As result during commit all metadata 
      locks were released (including lock for open HANDLER or
      global metadata shared lock). Indeed, attempt to release
      metadata lock for the second time which happened during
      HANLDER CLOSE or during release of GLR caused crash.
      
      This patch fixes problem by changing MDL_context's
      set_trans_sentinel() method to set sentinel to correct 
      value (it should point to the most recent ticket).
     @ mysql-test/include/handler.inc
        Added test for bug #51136 "Crash in pthread_rwlock_rdlock on 
        TEMPORARY + HANDLER + LOCK + SP".
     @ mysql-test/r/flush.result
        Updated test results (see flush.test).
     @ mysql-test/r/handler_innodb.result
        Updated test results (see include/handler.inc).
     @ mysql-test/r/handler_myisam.result
        Updated test results (see include/handler.inc).
     @ mysql-test/t/flush.test
        Added additional coverage for bug #51136 "Crash in
        pthread_rwlock_rdlock on TEMPORARY + HANDLER + LOCK +
        SP".
     @ sql/mdl.h
        When setting new value of transactional sentinel use 
        pointer to the most recent ticket instead of value 
        returned by MDL_context::mdl_savepoint(). 
        This allows to handle correctly situation when the new 
        value of sentinel should be the same as its current value 
        (MDL_context::mdl_savepoint() returns NULL in this case).
[15 Feb 2010 14:01] Dmitry Lenev
Fix for this bug was pushed into mysql-next-4284 tree. Since it is not repeatable outside of this non-public tree there is nothing to document and so I am simply closing bug.
[16 Feb 2010 16:47] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100216101445-2ofzkh48aq2e0e8o) (version source revid:alik@sun.com-20100215140849-b9fal65nwvrzczh4) (merge vers: 6.0.14-alpha) (pib:16)
[16 Feb 2010 16:56] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100216101208-33qkfwdr0tep3pf2) (version source revid:alik@sun.com-20100215120405-o1osx2k0nme27tx9) (pib:16)
[6 Mar 2010 11:02] Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100306103849-hha31z2enhh7jwt3) (version source revid:vvaintroub@mysql.com-20100216221947-luyhph0txl2c5tc8) (merge vers: 5.5.99-m3) (pib:16)
[7 Mar 2010 0:58] Paul DuBois
No changelog entry needed.