Bug #49770 crash in my_mb_wc_utf8mb3 due to corrupt processlist
Submitted: 17 Dec 2009 13:43 Modified: 15 Jul 2011 12:04
Reporter: Philip Stoev Email Updates:
Status: Won't fix Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:6.0-backup OS:Any
Assigned to: Jon Olav Hauglid CPU Architecture:Any

[17 Dec 2009 13:43] Philip Stoev
Description:
When executing a concurrent workload involving CRESTORE, DDL and KILL, mysqld crashed as follows:

#2  0x000000000067269d in handle_segfault (sig=11) at mysqld.cc:2769
#3  <signal handler called>
#4  0x0000000000a5e100 in my_mb_wc_utf8mb3 (cs=0x1059f80, pwc=0x7fc86f636ee8, s=0x0, e=0x4 <Address 0x4 out of bounds>) at ctype-utf8.c:2103
#5  0x0000000000a4e493 in my_well_formed_len_mb (cs=0x1059f80, b=0x0, e=0x4 <Address 0x4 out of bounds>, pos=64, error=0x7fc86f636f7c) at ctype-mb.c:296
#6  0x000000000066a269 in well_formed_copy_nchars (to_cs=0x1059f80, to=0x7fc86077d24c "", to_length=192, from_cs=0x1059f80, from=0x0, from_length=4,
    nchars=64, well_formed_error_pos=0x7fc86f637038, cannot_convert_error_pos=0x7fc86f637030, from_end_pos=0x7fc86f637028) at sql_string.cc:1012
#7  0x0000000000643e7c in Field_varstring::store (this=0x7fc86005d218, from=0x0, length=4, cs=0x1059f80) at field.cc:6845
#8  0x000000000082364c in fill_schema_processlist (thd=0x7fc86461a6b8, tables=0x7fc8600fa5a0, cond=0x0) at sql_show.cc:1999
#9  0x00000000008156c1 in get_schema_tables_result (join=0x7fc860a279b0, executed_place=PROCESSED_BY_JOIN_EXEC) at sql_show.cc:6643
#10 0x000000000072a9f7 in JOIN::exec (this=0x7fc860a279b0) at sql_select.cc:2434
#11 0x000000000061c468 in subselect_single_select_engine::exec (this=0x7fc8600fac38) at item_subselect.cc:2310
#12 0x00000000006212ce in Item_subselect::exec (this=0x7fc8600fab48) at item_subselect.cc:283
#13 0x0000000000619057 in Item_singlerow_subselect::val_int (this=0x7fc8600fab48) at item_subselect.cc:632
#14 0x00000000005d7f0a in Arg_comparator::compare_int_signed (this=0x7fc8600fad28) at item_cmpfunc.cc:1271
#15 0x00000000005e8684 in Arg_comparator::compare (this=0x7fc8600fad28) at item_cmpfunc.h:74
#16 0x00000000005dc479 in Item_func_lt::val_int (this=0x7fc8600fac78) at item_cmpfunc.cc:1771
#17 0x00000000005be4da in eval_const_cond (cond=0x7fc8600fac78) at item_func.cc:64
#18 0x000000000070ce4e in remove_eq_conds (thd=0x7fc86461a6b8, cond=0x7fc8600fac78, cond_value=0x7fc86f637950) at sql_select.cc:13676
#19 0x000000000070c82e in remove_eq_conds (thd=0x7fc86461a6b8, cond=0x7fc8600f9700, cond_value=0x7fc860a27790) at sql_select.cc:13547
#20 0x0000000000711058 in optimize_cond (join=0x7fc860a21be0, conds=0x7fc8600f9700, join_list=0x7fc86461c830, build_equalities=true,
    cond_value=0x7fc860a27790) at sql_select.cc:13514
#21 0x0000000000721cd8 in JOIN::optimize (this=0x7fc860a21be0) at sql_select.cc:1489
#22 0x0000000000727084 in mysql_select (thd=0x7fc86461a6b8, rref_pointer_array=0x7fc86461c8b0, tables=0x36f2b00, wild_num=0, fields=@0x7fc86461c7d0,
    conds=0x7fc8600f9700, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2685159936, result=0x36f2a28, unit=0x7fc86461c038,
    select_lex=0x7fc86461c6c8) at sql_select.cc:3106
#23 0x000000000072cb61 in handle_select (thd=0x7fc86461a6b8, lex=0x7fc86461bf98, result=0x36f2a28, setup_tables_done_option=0) at sql_select.cc:307
#24 0x000000000068387b in execute_sqlcom_select (thd=0x7fc86461a6b8, all_tables=0x36f2b00) at sql_parse.cc:4971
#25 0x0000000000685247 in mysql_execute_command (thd=0x7fc86461a6b8) at sql_parse.cc:2160
#26 0x000000000068d787 in mysql_parse (thd=0x7fc86461a6b8,
    inBuf=0x36f2420 "SELECT MAX(id) INTO @kill_id FROM information_schema.processlist WHERE id < CONNECTION_ID() AND (INFO LIKE CONCAT('%',TRIM(' testdb_S '),'%') OR INFO LIKE CONCAT('%',TRIM(' testdb_N '),'%')) AND 10 + "..., length=356, found_semicolon=0x7fc86f639f00) at sql_parse.cc:5985
#27 0x000000000068e320 in dispatch_command (command=COM_QUERY, thd=0x7fc86461a6b8,
    packet=0x7fc8604395e9 "SELECT MAX(id) INTO @kill_id FROM information_schema.processlist WHERE id < CONNECTION_ID() AND (INFO LIKE CONCAT('%',TRIM(' testdb_S '),'%') OR INFO LIKE CONCAT('%',TRIM(' testdb_N '),'%')) AND 10 + "..., packet_length=357) at sql_parse.cc:1078
#28 0x000000000068f892 in do_command (thd=0x7fc86461a6b8) at sql_parse.cc:760
#29 0x000000000067c6b0 in handle_one_connection (arg=0x7fc86461a6b8) at sql_connect.cc:1164
#30 0x000000315b0073da in start_thread () from /lib64/libpthread.so.0
#31 0x000000315a4e627d in clone () from /lib64/libc.so.6

This is because in fill_schema_processlist we have:

#8  0x000000000082364c in fill_schema_processlist (thd=0x7fc86461a6b8, tables=0x7fc8600fa5a0, cond=0x0) at sql_show.cc:1999
1999            table->field[3]->store(tmp->db, strlen(tmp->db), cs);
(gdb) list
1994            table->field[2]->store(tmp_sctx->host_or_ip,
1995                                   strlen(tmp_sctx->host_or_ip), cs);
1996          /* DB */
1997          if (tmp->db)
1998          {
1999            table->field[3]->store(tmp->db, strlen(tmp->db), cs);
2000            table->field[3]->set_notnull();
2001          }
2002
2003          pthread_mutex_lock(&tmp->LOCK_thd_data);
(gdb) print tmp->db
$10 = 0x0

which is a contradiction since on line 1997 we check that tmp->db is not zero. The only way for this situation to form is that the data structure is being accessed by several threads at the same time. Unfortunately, it looks like all participating threads are properly try to obtain LOCK_thread_count and there are no two threads operating on the processlist at the same time.

How to repeat:
If this is repeatable, a test case will be provided. In the meantime, the core and the binary will be uploaded.
[17 Dec 2009 13:51] Philip Stoev
Code and binary:

http://mysql-systemqa.s3.amazonaws.com/var-bug49768.zip

Source:

revision-id: charles.bell@sun.com-20091214143531-eurme91wrjgcjwcw
date: 2009-12-14 09:35:31 -0500
build-date: 2009-12-17 15:51:31 +0200
revno: 2908
branch-nick: mysql-6.0-backup
[15 May 2010 8:57] Philip Stoev
Setting this to Verified in the hope that it can be fixed based on core file or code analysis.
[15 Jul 2011 12:04] Jon Olav Hauglid
Bug only reported against a now defunct tree. Closing as Won't fix.