Bug #39550 Mysql server crash for backup if view created from consecutive views.
Submitted: 19 Sep 2008 22:55 Modified: 15 Dec 2008 13:19
Reporter: Hema Sridharan Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Backup Severity:S3 (Non-critical)
Version:mysql-6.0-backup OS:Linux
Assigned to: Jørgen Løland CPU Architecture:Any

[19 Sep 2008 22:55] Hema Sridharan
Description:
1) Create database(v) and table(t1).
2) Create view from table t1.
3) Create views based on view in consecutive manner till next 11 views are created
4) At the 11th view, perform backup database operation. This results in mysql server crash.

create database v;
 use v;
 create table t1(id int);
 create view v1 as select * from t1;
 create view v2 as select * from v1;
 create view v3 as select * from v2;
 create view v4 as select * from v3;
 create view v5 as select * from v4;
 create view v6 as select * from v5;
 create view v7 as select * from v6;
 create view v8 as select * from v7;
 create view v9 as select * from v8;
 backup database v to 'v.bak';
 create view v10 as select * from v9;
 backup database v to 'v1.bak';
 create view v11 as select * from v10;
 backup database v to 'v2.bak';
 ERROR 2013 (HY000): Lost connection to MySQL server during query

How to repeat:
mysql> create database v;
Query OK, 1 row affected (0.00 sec)

mysql>  use v;
 create table t1(id int);
Database changed
mysql>  create table t1(id int);
mysql>  create view v1 as select * from t1;
mysql>  create view v2 as select * from v1;
mysql>  create view v3 as select * from v2;
mysql>  create view v4 as select * from v3;
mysql>  create view v5 as select * from v4;
mysql>  create view v6 as select * from v5;
mysql> create view v7 as select * from v6;
mysql>  create view v8 as select * from v7;
mysql>  create view v9 as select * from v8;
mysql>  backup database v to 'v.bak';
ý+-----------+
| backup_id |
+-----------+
| 253       |
+-----------+
1 row in set (0.65 sec)

mysql> create view v10 as select * from v9;
Query OK, 0 rows affected (0.00 sec)

mysql>  backup database v to 'v1.bak';
þ+-----------+
| backup_id |
+-----------+
| 254       |
+-----------+
1 row in set (0.67 sec)

mysql>  create view v11 as select * from v10;
Query OK, 0 rows affected (0.00 sec)

mysql> backup database v to 'v2.bak';
ERROR 2013 (HY000): Lost connection to MySQL server during query

STACK DUMP
==========

080919 22:29:01 [Note] Backup: Backing up selected databases
/export/home/tmp/rpl/backupmovep/libexec/mysqld(my_print_stacktrace+0x32)[0xd5e0ae]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(handle_segfault+0x2a6)[0x706c20]
/lib64/libpthread.so.0[0x31d720de60]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z27create_tmp_field_from_fieldP3THDP5FieldPKcP8st_tableP10Item_fieldj+0xd3)[0x78b42f]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z16create_tmp_fieldP3THDP8st_tableP4ItemNS3_4TypeEPPS4_PP5FieldSA_bbbbj+0x35e)[0x78e3a8]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z16create_tmp_tableP3THDP15TMP_TABLE_PARAMR4ListI4ItemEP8st_orderbbyyPc+0x10c8)[0x78f6be]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_ZN12select_union19create_result_tableEP3THDP4ListI4ItemEbyPKcb+0xc4)[0x8c8b3e]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z21mysql_derived_prepareP3THDP6st_lexP10TABLE_LIST+0x1a3)[0x8ca3bd]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z20mysql_handle_derivedP6st_lexPFbP3THDS0_P10TABLE_LISTE+0x80)[0x8c9f54]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x7b)[0x76d48f]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_ZN3obs8TableObj12do_serializeEP3THDP6String+0x1ed)[0x916d97]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_ZN3obs3Obj9serializeEP3THDP6String+0x4e)[0xdf8090]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(bcat_get_item_create_query+0x22d)[0xdf49ab]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(bstream_wr_item_def+0x55)[0xe0546f]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(bstream_wr_meta_data+0x235)[0xe04f42]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(bstream_wr_preamble+0x74)[0xe0390b]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_ZN6backup14write_preambleERKNS_10Image_infoERNS_13Output_streamE+0x2d)[0xdfb649]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_ZN18Backup_restore_ctx9do_backupEv+0x1b8)[0xdf6530]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z22execute_backup_commandP3THDP6st_lexP6String+0x406)[0xdf7d08]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z21mysql_execute_commandP3THD+0xb76)[0x71909a]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x275)[0x7213f3]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x99b)[0x721f1f]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(_Z10do_commandP3THD+0x220)[0x723414]
/export/home/tmp/rpl/backupmovep/libexec/mysqld(handle_one_connection+0x11a)[0x70fb30]
/lib64/libpthread.so.0[0x31d72062e7]
/lib64/libc.so.6(clone+0x6d)[0x31d66ce3bd]
080919 22:29:02 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60646 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x2aaaac8c69c8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4c5c30f0 thread_stack 0x40000
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xbd33c80 = backup database v to 'v2.bak'
thd->thread_id=1
thd->killed=NOT_KILLED

Suggested fix:
[20 Oct 2008 11:05] Jørgen Løland
The suggested test script does not fail anymore, but I am able to get the error if I add more views. Adding 2 more views before the first backup reproduces the crash.
[20 Oct 2008 11:08] Jørgen Løland
I experimented a bit with commenting out backup statements and got some strange results:

create database v;
use v;
create table t1(id int);

create view v1 as select * from t1;
create view v2 as select * from v1;
(...)
create view v14 as select * from v13;

# --echo Backup no 1
# --replace_column 1 #
# backup database v to 'v1.bak';

create view v15 as select * from v14;

# --echo Backup no 2
# --replace_column 1 #
# backup database v to 'v2.bak';

create view v16 as select * from v15;
# create view v17 as select * from v16;
# create view v18 as select * from v17;
# create view v19 as select * from v18;

--echo Backup no 3
--replace_column 1 #
backup database v to 'v3.bak';

----------------------------------

With view v16:
* Only backup #3:       passed
* Backup #1 and #3:     passed
* Backup #2 and #3:     passed
* Backup #1, #2 and #3: crash on #3

With view v17:
* Only backup #3:       crash on #3
* Backup #1 and #3:     passed
* Backup #2 and #3:     passed
* Backup #1, #2 and #3: passed

With view v18:
* Only backup #3:       crash on #3
* Backup #1 and #3:     crash on #3
* Backup #2 and #3:     passed
* Backup #1, #2 and #3: passed

With view v19:
* Only backup #3:       crash on #3
* Backup #1 and #3:     crash on #3
* Backup #2 and #3:     crash on #3
* Backup #1, #2 and #3: crash on #3
[17 Nov 2008 11:58] Jørgen Løland
Update with latest info:

*Stack trace from gdb when the error occurs:*
-----------
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xaf8ea790 (LWP 25466)]
0x082161c4 in Field::maybe_null (this=0x962d0b8) at ../../sql/field.h:283
283       inline bool maybe_null(void) { return null_ptr != 0 || table->maybe_null; }
(gdb) bt
#0  0x082161c4 in Field::maybe_null (this=0x962d0b8) at ../../sql/field.h:283
#1  0x0836d3e1 in create_tmp_field (thd=0x9567830, table=0x9964a58,
    item=0x996a7a0, type=Item::FIELD_ITEM, copy_func=0xaf8e8450,
    from_field=0x9965360, default_field=0x9965350, group=false,
    modify_item=false, table_cant_handle_bit_fields=false,
    make_copy_field=false, convert_blob_length=0) at sql_select.cc:11523
#2  0x0836e807 in create_tmp_table (thd=0x9567830, param=0x99881c4,
    fields=@0x996a0f8, group=0x0, distinct=false, save_sum_fields=true,
    select_options=2416204288, rows_limit=18446744073709551615,
    table_alias=0x99698b0 "v5") at sql_select.cc:11944
#3  0x084a3f46 in select_union::create_result_table (this=0x99881b8,
    thd_arg=0x9567830, column_types=0x996a0f8, is_union_distinct=false,
    options=2416204288, table_alias=0x99698b0 "v5", bit_fields_as_long=false) at sql_union.cc:129
#4  0x084a56a6 in mysql_derived_prepare (thd=0x9567830, lex=0x956894c,
    orig_table_list=0x99698b8) at sql_derived.cc:139
#5  0x084a5247 in mysql_handle_derived (lex=0x956894c,
    processor=0x84a54fc <mysql_derived_prepare(THD*, LEX*, TABLE_LIST*)>) at sql_derived.cc:56
#6  0x08349a18 in open_normal_and_derived_tables (thd=0x9567830,
    tables=0x9978008, flags=0) at sql_base.cc:4163
#7  0x084f12be in obs::TableObj::do_serialize (this=0x975fef8, thd=0x9567830, serialization=0x95ebd64) at si_objects.cc:2130
------------
(gdb) p table
$1 = (TABLE *) 0x0

*I run the following test:*
---------------------------
create database v;
use v;
create table t1(id int);

create view v1 as select * from t1;
create view v2 as select * from v1;
create view v3 as select * from v2;
create view v4 as select * from v3;
create view v5 as select * from v4;
create view v6 as select * from v5;
create view v7 as select * from v6;
create view v8 as select * from v7;
create view v9 as select * from v8;
create view v10 as select * from v9;
create view v11 as select * from v10;
create view v12 as select * from v11;
create view v13 as select * from v12;
create view v14 as select * from v13;
create view v15 as select * from v14;
create view v16 as select * from v15;
create view v17 as select * from v16;

*And the following script in gdb:*
----------------------------------
b kernel.cc:195
run

b stream_v1.c:1258
c
c 6
b sql_select.cc:11928
c
c 9
# how field looks like when it works
p *((Item_field*)item)->field
p ((Item_field*)item)->field->table
c
# how field looks like when it crashes
p *((Item_field*)item)->field
p ((Item_field*)item)->field->table

*I get the following output from this script*
---------------------------------------------
The one that works:
(gdb) p *((Item_field*)item)->field
$2 = {_vptr.Field = 0x893eb88, ptr = 0x9962a69 "", null_ptr = 0x9962a68 "ÿ",
  table = 0x95f59c8, orig_table = 0x95bacb8, table_name = 0x95f5a48,
  field_name = 0x997cee8 "id", comment = {str = 0x89a02b7 "", length = 0},
  key_start = {map = 0}, part_of_key = {map = 0}, part_of_key_not_clustered = {map = 0}, part_of_sortkey = {map = 0}, part_of_key_wo_keyread = {map = 0},
  unireg_check = Field::NONE, field_length = 11, flags = 0, field_index = 0,
  null_bit = 2 '\002', is_created_from_null_item = false}
(gdb) p ((Item_field*)item)->field->table

The one that crashes:
(gdb) p *((Item_field*)item)->field
$5 = {_vptr.Field = 0x0, ptr = 0x3333 <Address 0x3333 out of bounds>,
  null_ptr = 0x0, table = 0x0, orig_table = 0x199980, table_name = 0x0,
  field_name = 0x0, comment = {
    str = 0xcccc000 <Address 0xcccc000 out of bounds>, length = 0},
  key_start = {map = 7376896189632872448}, part_of_key = {map = 0},
  part_of_key_not_clustered = {map = 34359751475}, part_of_sortkey = {
    map = 676334066378735616}, part_of_key_wo_keyread = {
    map = 562949953421312}, unireg_check = Field::NONE, field_length = 0,
  flags = 0, field_index = 0, null_bit = 0 '\0',
  is_created_from_null_item = false}

So, field has not been set properly in this case, or has been overwritten. A strange thing is that if I change the view names from v<#> to vvvv<#> (or any other >3 chars before the id), the bug goes away. That makes me think that the memory location is overwritten somewhere.
[5 Dec 2008 13:18] Jørgen Løland
To figure out where the uninitialized field comes from, I added Item iterators to the code run just before the crash. E.g., in sql_derived.cc:mysql_derived_prepare:

bool mysql_derived_prepare(THD *thd, LEX *lex, TABLE_LIST *orig_table_list)
{
  SELECT_LEX_UNIT *unit= orig_table_list->derived;

  ulonglong create_options;
  DBUG_ENTER("mysql_derived_prepare");
  bool res= FALSE;

  if (unit)
  {
    // ADDED CODE STARTS HERE
    List_iterator_fast<Item> lst(unit->types);
    Item *itm;
    int myint=0;
    while ((itm=lst++)) {
      myint++;
    }
  ...

I find the uninitialized Item at this level as well.
[5 Dec 2008 13:19] Jørgen Løland
Tried to debug on another machine. The bug still appears, but not for the same Item.
[15 Dec 2008 13:19] Jørgen Løland
The new server interface (si_object) has been pushed to the backup tree. I am no longer able to reproduce this bug, even if I create as much as 50.
[15 Dec 2008 13:21] Jørgen Løland
Likely fixed by wl#4264, which was pushed to 6.0.9 December 8.