Bug #52856 concurrent show columns or show full columns causes a crash!!!1
Submitted: 15 Apr 2010 10:10 Modified: 7 Jul 2010 19:47
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Locking Severity:S1 (Critical)
Version:5.5.3,5.5.4, 5.6.99 OS:Any (XP64, linux)
Assigned to: Sergei Glukhov CPU Architecture:Any
Tags: crash, regression, show full columns

[15 Apr 2010 10:10] Shane Bester
Description:
lock violation or crash during concurrent execution of a select and show full columns from a table:
5.5.3 stack trace:

 Field_varstring::sql_type()[field.cc:7065]
 store_column_type()[sql_show.cc:3895]
 get_schema_column_record()[sql_show.cc:4118]
 fill_schema_show_cols_or_idxs()[sql_show.cc:2964]
 get_all_tables()[sql_show.cc:3382]
 get_schema_tables_result()[sql_show.cc:6643]
 JOIN::exec()[sql_select.cc:1830]
 mysql_select()[sql_select.cc:2508]
 handle_select()[sql_select.cc:271]
 execute_sqlcom_select()[sql_parse.cc:4703]
 mysql_execute_command()[sql_parse.cc:2191]
 mysql_parse()[sql_parse.cc:5740]
 dispatch_command()[sql_parse.cc:1026]
 do_command()[sql_parse.cc:710]
 do_handle_one_connection()[sql_connect.cc:1174]
 handle_one_connection()[sql_connect.cc:1115]
 pthread_start()[my_winthread.c:63]
 _callthreadstartex()[threadex.c:348]
 _threadstartex()[threadex.c:326]
 BaseThreadStart()

in fact with one test, the "select" thread was selecting from the table
that the other connection had write locked. this was harder to repeat though.

How to repeat:
it is hard to repeat.
start a very fast server with --no-defaults --skip-gra --skip-na

drop table if exists `t1`;
create table `t1`(`a` char(255)) engine=myisam default charset utf8;

in 2 or more threads, run this continuously

show full columns from t1;

Suggested fix:
fix locking problems.
[15 Apr 2010 17:43] Sveta Smirnova
Thank you for the report.

Crash verified with different trace:

stack_bottom = 0x4826f078 thread_stack 0x20000
./sbin/mysqld(my_print_stacktrace+0x35)[0x9fcecc]
./sbin/mysqld(handle_segfault+0x2a7)[0x51dc98]
/lib64/libpthread.so.0[0x3429e0dd40]
./sbin/mysqld(_ZNK12Field_string8sql_typeER6String+0x15a)[0x6f40c6]
./sbin/mysqld[0x618ed3]
./sbin/mysqld[0x61535c]
./sbin/mysqld(_Z14get_all_tablesP3THDP10TABLE_LISTP4Item+0x246)[0x61605f]
./sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x2f4)[0x623fe1]
./sbin/mysqld(_ZN4JOIN4execEv+0x6da)[0x5e04e0]
./sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x2f9)[0x5e298f]
./sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1a4)[0x5dad3a]
./sbin/mysqld[0x5b817c]
./sbin/mysqld(_Z21mysql_execute_commandP3THD+0x966)[0x5b07f2]
./sbin/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x280)[0x5ba283]
./sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xb08)[0x5adfd0]
./sbin/mysqld(_Z10do_commandP3THD+0x252)[0x5ad2b3]
./sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x14d)[0x68df2d]
./sbin/mysqld(handle_one_connection+0x33)[0x68ddd9]
./sbin/mysqld(_Z16pfs_spawn_threadPv+0xb4)[0x9d042c]
/lib64/libpthread.so.0[0x3429e061b5]
/lib64/libc.so.6(clone+0x6d)[0x34292cd39d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x4a6c2f8 = show full columns from t1
thd->thread_id=41176
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

To repeat run `while true; do ./bin/mysql -uroot -S /tmp/mysql_ssmirnova.sock test -e "show full columns from t1"; done` in 2 terminals
[16 Apr 2010 12:01] Jon Olav Hauglid
The crash is repeatable by adding the following sync point:

=== modified file 'sql/sql_show.cc'
--- sql/sql_show.cc	2010-04-14 07:40:45 +0000
+++ sql/sql_show.cc	2010-04-16 11:53:49 +0000
@@ -17,6 +17,7 @@
 /* Function with list databases, tables or fields */
 
 #include "mysql_priv.h"
+#include "debug_sync.h"
 #include "sql_select.h"                         // For select_describe
 #include "sql_show.h"
 #include "repl_failsafe.h"
@@ -4068,6 +4069,7 @@ static int get_schema_column_record(THD 
     /* to satisfy 'field->val_str' ASSERTs */
     field->table= show_table;
     show_table->in_use= thd;
+    DEBUG_SYNC(thd, "get_schema_column");
 
     if (wild && wild[0] &&
         wild_case_compare(system_charset_info, field->field_name,wild))

and then running the following MTR test case:

--source include/have_debug_sync.inc
CREATE TABLE t1(a CHAR(255));

connect(con1, localhost, root);
SET DEBUG_SYNC= "get_schema_column SIGNAL waiting WAIT_FOR completed";
--send SHOW FULL COLUMNS FROM t1

connection default;
SET DEBUG_SYNC= "now WAIT_FOR waiting";
SHOW FULL COLUMNS FROM t1;
SET DEBUG_SYNC= "now SIGNAL completed";

connection con1;
--reap
[27 Apr 2010 12:31] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/106655

3008 Jon Olav Hauglid	2010-04-27
      Bug #52856 concurrent show columns or show full columns causes a crash!!!1
      
      This crash could occur if two connections concurrenly tried to execute
      SHOW (FULL) COLUMNS. During processing of the statments, details about
      the table columns are used to populate an information schema table.
      The problem was that this included temporarily modifying table column
      data as this was needed by table column code. Since the two connections
      then modified the same data structures, it was possible to get a thread
      timing where one connection made table column data set by the other 
      connection, invalid.
      
      In more detail, the problem was that get_schema_column_record() 
      runs through all Fields in TABLE_SHARE::field to get column details.
      For each field, field->table (and field->table->in_use) are set as 
      they are needed inside the field code. The crash would occur if the
      the timing between two connections were as follows:
       con1: set field->table and table->in_use
       con2: set field->table and table->in_use
       con2: access field->table and table->in_use
       con2: close table and set table->in_use = NULL
       con1: access field->table and table->in_use => CRASH 
      
      This patch fixes the problem by making sure LOCK_open is taken before
      trying to populate the I_S table with data from TABLE_SHARE fields.
      This is similar to what is already done when executing the similar
      "SELECT * FROM information_schema.columns" statement.
      
      No test case added. A very exact timing of the execution of the two
      SHOW (FULL) COLUMNS statements were needed to trigger the error. 
      This was only possible to reliably reproduce using a sync point.
      See the bug page for details about the sync point and a possible test
      case. With this patch, the thread timing needed is no longer possible
      to achieve due to LOCK_open preventing two threads from modifying the
      TABLE_SHARE fields at the same time.
[5 May 2010 6:51] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/107404

3142 Sergey Glukhov	2010-05-05
      Bug#52856 concurrent show columns or show full columns causes a crash!!!1
      We should avoid any SHARE fields assignments as
      this is shared structure and assignments may
      affect other therads. To avoid this
      copy of SHARE struct is created and
      stored into TABLE struct which is
      used in get_schema_coulumns_record later.
     @ mysql-test/r/mdl_sync.result
        test case
     @ mysql-test/t/mdl_sync.test
        test case
     @ sql/sql_show.cc
        We should avoid any SHARE fields assignments as
        this is shared structure and assignments may
        affect other therads. To avoid this
        copy of SHARE struct is created and
        stored into TABLE struct which is
        used in get_schema_coulumns_record later.
[27 May 2010 15:54] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/109399

3068 Sergey Glukhov	2010-05-27
      Bug#52856 concurrent show columns or show full columns causes a crash!!!1
      We should avoid any SHARE fields assignments as
      this is shared structure and assignments may
      affect other therads. To avoid this
      copy of SHARE struct is created and
      stored into TABLE struct which is
      used in get_schema_coulumns_record later.
     @ mysql-test/r/mdl_sync.result
        test case
     @ mysql-test/t/mdl_sync.test
        test case
     @ sql/sql_show.cc
        We should avoid any SHARE fields assignments as
        this is shared structure and assignments may
        affect other therads. To avoid this
        copy of SHARE struct is created and
        stored into TABLE struct which is
        used in get_schema_coulumns_record later.
[15 Jun 2010 8:17] Bugs System
Pushed into 5.5.5-m3 (revid:alik@sun.com-20100615080459-smuswd9ooeywcxuc) (version source revid:mmakela@bk-internal.mysql.com-20100415070122-1nxji8ym4mao13ao) (merge vers: 5.1.47) (pib:16)
[15 Jun 2010 8:34] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100615080558-cw01bzdqr1bdmmec) (version source revid:mmakela@bk-internal.mysql.com-20100415070122-1nxji8ym4mao13ao) (pib:16)
[7 Jul 2010 19:47] Paul DuBois
Noted in 5.5.5 changelog.

Concurrent SHOW COLUMNS statements could cause a server crash.