Bug #38363 Crash in open_normal_and_derived_tables() when called from si_objects
Submitted: 25 Jul 2008 8:03 Modified: 30 Mar 2009 2:07
Reporter: Rafal Somla Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: General Severity:S3 (Non-critical)
Version:mysql-6.0-backup OS:Any
Assigned to: Hema Sridharan CPU Architecture:Any
Triage: Triaged: D1 (Critical)

[25 Jul 2008 8:03] Rafal Somla
Description:
This is a sporadic crash which Hema detected few times but which can not be repeated consistently.

Hema wrote:
> I am noticing Mysql server crash sometimes in the latest backup team tree
> while performing backup.
>  
> Note: This problem does not  happen every time when I ran the test and I was
> not exactly able to figure out the reason for this crash. 

The crash happened when she run backup.backup_special_characters test on one of ndb hosts. MTR reports:

 backup.backup_special_characters 'innodb' [ fail ]

 mysqltest: At line 100: query 'BACKUP DATABASE `sp!`, `sp@` to 'sp.bak'' failed: 2013: Lost connection to MySQL server during query

and produces the following call stack:

CURRENT_TEST: backup.backup_special_characters
080723  1:32:57  InnoDB: Started; log sequence number 0 46419
080723  1:32:57 [Note] Event Scheduler: Loaded 0 events
080723  1:32:57 [Note] /data0/team6.0/mysql-6.0-backup/sql/mysqld: ready for connections.
Version: '6.0.6-alpha-debug-log'  socket: '/data0/team6.0/mysql-6.0-backup/mysql-test/var/tmp/master.sock'  port: 9306  Source distribution
080723  1:32:58 [Note] Backup: Starting backup process
080723  1:32:58 [Note] Backup: Backing up selected databases
/data0/team6.0/mysql-6.0-backup/sql/mysqld(my_print_stacktrace+0x2f)[0xce029f]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(handle_segfault+0x26f)[0x6eff57]
/lib64/tls/libpthread.so.0[0x3d6a30c430]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z27create_tmp_field_from_fieldP3THDP5FieldPKcP8st_tableP10Item_fieldj+0xd7)[0x7759b3]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z16create_tmp_fieldP3THDP8st_tableP4ItemNS3_4TypeEPPS4_PP5FieldSA_bbbbj+0x2d7)[0x7762ff]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z16create_tmp_tableP3THDP15TMP_TABLE_PARAMR4ListI4ItemEP8st_orderbbyyPc+0xef3)[0x777533]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_ZN12select_union19create_result_tableEP3THDP4ListI4ItemEbyPKcb+0xc7)[0x8884b1]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z21mysql_derived_prepareP3THDP6st_lexP10TABLE_LIST+0x196)[0x88a924]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z20mysql_handle_derivedP6st_lexPFbP3THDS0_P10TABLE_LISTE+0x89)[0x88a715]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x7b)[0x746f43]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_ZN3obs8TableObj12do_serializeEP3THDP6String+0x208)[0x8d236e]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_ZN3obs3Obj9serializeEP3THDP6String+0x4e)[0xd728e2]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(bcat_get_item_create_query+0x209)[0xd70d7b]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(bstream_wr_item_def+0x35)[0xd7f6a7]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(bstream_wr_meta_data+0x258)[0xd7f189]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(bstream_wr_preamble+0x7c)[0xd7db57]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_ZN6backup14write_preambleERKNS_10Image_infoERNS_13Output_streamE+0x2d)[0xd71d37]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_ZN18Backup_restore_ctx9do_backupEv+0x176)[0xd6f01e]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z22execute_backup_commandP3THDP6st_lex+0x32c)[0xd6db3c]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z21mysql_execute_commandP3THD+0x77f)[0x7009fb]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x1b4)[0x7096da]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x8e2)[0x6fea42]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(_Z10do_commandP3THD+0x206)[0x6fdf6e]
/data0/team6.0/mysql-6.0-backup/sql/mysqld(handle_one_connection+0x128)[0x6fbb48]
/lib64/tls/libpthread.so.0[0x3d6a30610a]
/lib64/tls/libc.so.6(__clone+0x73)[0x3d69ac68b3]
080723  1:32:59 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=3
max_threads=151
thread_count=3
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60646 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x25fe498
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4c9171b8 thread_stack 0x40000
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x260a000 = BACKUP DATABASE `sp!`, `sp@` to 'sp.bak'
thd->thread_id=3
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

I examined core files with gdb, an overwiev will follow.

How to repeat:
Hema got this randomly when running backup_special_characters test on ndb host.
[25 Jul 2008 8:04] Rafal Somla
The call stack at crash time as reported by gdb: 

> #3  <signal handler called>
> #4  0x00000000007759b3 in create_tmp_field_from_field (thd=0x25fe498, org_field=0x263e570, name=0x279b650 "$$", 
>     table=0x26ca420, item=0x0, convert_blob_length=0) at sql_select.cc:11203
> #5  0x00000000007762ff in create_tmp_field (thd=0x25fe498, table=0x26ca420, item=0x279b550, type=Item::FIELD_ITEM, 
>     copy_func=0x4c914ce0, from_field=0x26cb068, default_field=0x26cb058, group=false, modify_item=false, 
>     table_cant_handle_bit_fields=false, make_copy_field=false, convert_blob_length=0) at sql_select.cc:11480
> #6  0x0000000000777533 in create_tmp_table (thd=0x25fe498, param=0x27a85d8, fields=@0x279a9f0, group=0x0, 
>     distinct=false, save_sum_fields=true, select_options=2416204288, rows_limit=18446744073709551615, 
>     table_alias=0x2799df8 "v2%&") at sql_select.cc:11881
> #7  0x00000000008884b1 in select_union::create_result_table (this=0x27a85c0, thd_arg=0x25fe498, column_types=0x279a9f0, 
>     is_union_distinct=false, options=2416204288, table_alias=0x2799df8 "v2%&", bit_fields_as_long=false)
>     at sql_union.cc:129
> #8  0x000000000088a924 in mysql_derived_prepare (thd=0x25fe498, lex=0x2600018, orig_table_list=0x2799e00)
>     at sql_derived.cc:139
> #9  0x000000000088a715 in mysql_handle_derived (lex=0x2600018, 
>     processor=0x88a78e <mysql_derived_prepare(THD*, st_lex*, TABLE_LIST*)>) at sql_derived.cc:56
> #10 0x0000000000746f43 in open_normal_and_derived_tables (thd=0x25fe498, tables=0x27a2ff0, flags=0) at sql_base.cc:4153
> #11 0x00000000008d236e in obs::TableObj::do_serialize (this=0x26cd270, thd=0x25fe498, serialization=0x269e930)
>     at si_objects.cc:1772
> #12 0x0000000000d728e2 in obs::Obj::serialize (this=0x26cd270, thd=0x25fe498, serialization=0x269e930)
>     at ../si_objects.h:188
> #13 0x0000000000d70d7b in bcat_get_item_create_query (catalogue=0x269aeb8, item=0x269f018, stmt=0x4c9155c0)
>     at kernel.cc:1752
> #14 0x0000000000d7f6a7 in bstream_wr_item_def (s=0x2674938, cat=0x269aeb8, kind=PER_DB_ITEM, item=0x269f018)
>     at stream_v1.c:1558
> #15 0x0000000000d7f189 in bstream_wr_meta_data (s=0x2674938, cat=0x269aeb8) at stream_v1.c:1258
> #16 0x0000000000d7db57 in bstream_wr_preamble (s=0x2674938, hdr=0x269aeb8) at stream_v1.c:164
> #17 0x0000000000d71d37 in backup::write_preamble (info=@0x269aeb0, s=@0x2674930) at stream.h:156
> #18 0x0000000000d6f01e in Backup_restore_ctx::do_backup (this=0x4c915750) at kernel.cc:834
> #19 0x0000000000d6db3c in execute_backup_command (thd=0x25fe498, lex=0x2600018) at kernel.cc:178
> #20 0x00000000007009fb in mysql_execute_command (thd=0x25fe498) at sql_parse.cc:2172
> #21 0x00000000007096da in mysql_parse (thd=0x25fe498, inBuf=0x260a000 "BACKUP DATABASE `sp!`, `sp@` to 'sp.bak'", 
>     length=40, found_semicolon=0x4c916ed8) at sql_parse.cc:5800

The place of crash is:

> 11198       new_field= new Field_varstring(convert_blob_length,
> 11199                                      org_field->maybe_null(),
> 11200                                      org_field->field_name, table->s,
> 11201                                      org_field->charset());
> 11202     else
> 11203       new_field= org_field->new_field(thd->mem_root, table,
> 11204                                       table == org_field->table);
> 11205     if (new_field)
> 11206     {
> 11207       new_field->init(table);

Some values from the place of crash:

> (gdb) p org_field
> $1 = (class Field *) 0x263e570
> (gdb) p thd->mem_root
> $2 = (MEM_ROOT *) 0x26cabf0
> (gdb) p table
> $3 = (TABLE *) 0x26ca420
> (gdb) p org_field->table
> $4 = (st_table *) 0x0
> (gdb) p *org_field
> $7 = {_vptr.Field = 0x0, ptr = 0x0, null_ptr = 0x1111 <Address 0x1111 out of bounds>, table = 0x0, orig_table = 0x0, 
>   table_name = 0x88880, field_name = 0x0, comment = {str = 0x0, length = 71581696}, key_start = {map = 0}, 
>   part_of_key = {map = 0}, part_of_key_not_clustered = {map = 9162457088}, part_of_sortkey = {map = 0}, 
>   part_of_key_wo_keyread = {map = 18764712116224}, unireg_check = 24, field_length = 0, flags = 0, field_index = 0, 
>   null_bit = 0 '\0', is_created_from_null_item = false}

And the same situation from a second crash detected by Hema:

> (gdb) p *org_field
> $2 = {_vptr.Field = 0xffffffffffffffff, ptr = 0x0, 
>   null_ptr = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>, table = 0x2, orig_table = 0x0, 
>   table_name = 0x26958f0, field_name = 0x26958f0 "h4z\025\217\217\217\217ñ", comment = {str = 0x0, length = 0}, 
>   key_start = {map = 0}, part_of_key = {map = 0}, part_of_key_not_clustered = {map = 0}, part_of_sortkey = {map = 0}, 
>   part_of_key_wo_keyread = {map = 0}, unireg_check = Field::NONE, field_length = 0, flags = 0, field_index = 0, 
>   null_bit = 0 '\0', is_created_from_null_item = false}

It looks like org_field is a dangling pointer when execution reaches line 11203 of sql_select.cc. It is either a wrong logic in sql_select.cc or something is wrong in the way open_normal_and_derived_tables() is called from obs::TableObj::do_serialize in si_objects.cc.
[25 Jul 2008 8:10] Rafal Somla
The si_objects code which calls open_normal_and_derived_tables() is being rewritten now as WL#4264. We should wait for the changes and see if the problem remains after them.
[16 Sep 2008 1:37] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/54171

2698 Hema Sridharan	2008-09-16
      WL#4227. The following test cases fails sporadically because of bug#38363.
      After fixing the Bug#38363, these test cases should be pushed in to the tree.
[9 Oct 2008 19:33] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/55975

2703 Hema Sridharan	2008-10-09
      WL#4227: Changes made according to review comments of Chuck.
      Currently 2 tests are included backup_datatypes and backup_partitions
      backup_partitions is kept in backup_engines suite and will be executed
      against all storage engines. backup_datatypes test is in backup suite.
      There are seperate test cases for special characters and accented characters for this WL#
      Currently these tests are failing because of Bug#38363, so attached them in them in the bug report.
[11 Dec 2008 13:29] Alexander Nozdrin
The bug should be fixed by a patch for WL#4264 pushed into 6.0.8-alpha.
[26 Dec 2008 8:15] Alexander Nozdrin
The bug is fixed. Assigning to Hema to add missing test cases.
[22 Jan 2009 23:47] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/63854

2754 Hema Sridharan	2009-01-23
      Bug#38363. Test cases for this bug is committed.
      Tests are failing sporadically. 
      backup_special_characters and backup_accented 
      are two tests included in backup_engines suite
[22 Jan 2009 23:51] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/63855

2970 Hema Sridharan	2009-01-23
      Bug#38363. Test cases backup_special_characters and backup_accented are failing sporadically because of bug#38363.
      These tests are disabled in main tree(mysql-6.0)
[2 Feb 2009 23:08] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/64952

2756 Hema Sridharan	2009-02-03
      Bug#38363 ( Two tests committed for this bug,
      backup_special_characters and backup_accented.test)
[3 Feb 2009 19:39] Chuck Bell
Patch approved.
[4 Feb 2009 14:26] Jørgen Løland
Patch approved
[4 Feb 2009 16:19] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65197

2756 Hema Sridharan	2009-02-04
      BUG#38363
[4 Feb 2009 18:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/65219

2757 Hema Sridharan	2009-02-04
      BUG#38363
[26 Mar 2009 12:34] Bugs System
Pushed into 6.0.11-alpha (revid:alik@sun.com-20090326121822-pt84kzxxayzho4mn) (version source revid:hema@mysql.com-20090204180632-gyulqbv1gf7pan4a) (merge vers: 6.0.10-alpha) (pib:6)
[30 Mar 2009 2:07] Paul Dubois
Test case changes. No changelog entry needed.