Bug #41367 Server crash from 6.0.7-alpha
Submitted: 10 Dec 2008 17:58 Modified: 14 Sep 2009 10:13
Reporter: Akiko Marti Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:6.0.7-alpha, 6.0.8 OS:Solaris
Assigned to: CPU Architecture:Any

[10 Dec 2008 17:58] Akiko Marti
Description:
The following delete query (RF2 from DBT3) runs in 0.62 sec under MySQL 5.1.29 but crashes the Server from 6.0.7-alpha. Both were compiled with Sun Studio 12 and run on the same system using the same my.cnf file. They produce the same explain plan.

Query: delete lineitem from lineitem inner join tmp_orderkey1 on
        l_orderkey=tmp_orderkey1.orderkey; 

InnoDB tables: 
        lineitem ~6 million rows, tmp_orderkey1 1500 rows
        the query deletes ~6,000 rows (0.1%) from lineitem

MySQL 5.1.29-rc:   0.62 sec
MySQL 6.0.7-alpha: Server crash

Source distribution for 5.1.29-rc and 6.0.7-alpha were compiled with Sun Studio 12 Sun C/C++ 5.9 

err file:

/usr/local/mysql607/libexec/mysqld'my_print_stacktrace+0x21 [0xb25781]
/lib/amd64/libc.so.1'call_user_handler+0x2a7 [0xfffffd7ffed3d4df]
/usr/local/mysql607/libexec/mysqld'row_sel_field_store_in_mysql_format+0x29c [0xa96d1c]
081210  9:33:32 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=14
max_threads=100
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6575624 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x31e4e15a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 3000003d832 is an invalid pointer
thd->thread_id=12321848856440699817
thd->killed=UNKNOWN
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
081210 09:50:44 mysqld_safe mysqld restarted
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
081210  9:50:50  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
081210  9:50:52  InnoDB: Started; log sequence number 1 2006701681
081210  9:50:52 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '6.0.7-alpha'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution

Explain plan for select is the same between 5.1.29 and 6.0.7-alpha (I don't think I can produce the plan for delete):

mysql> explain select count(*) from lineitem inner join tmp_orderkey1 on l_orderkey=tmp_orderkey1.orderkey\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: tmp_orderkey1
         type: index
possible_keys: i_tmp_okey
          key: i_tmp_okey
      key_len: 6
          ref: NULL
         rows: 1524
        Extra: Using index
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: lineitem
         type: ref
possible_keys: PRIMARY,i_l_orderkey,i_l_orderkey_quantity
          key: i_l_orderkey
      key_len: 4
          ref: dbt3.tmp_orderkey1.orderkey
         rows: 1
        Extra: Using where; Using index 

How to repeat:
Install and run DBT3 benchmark (Scale Factor 1).
This delete query is a part of the DBT3 benchmark (Refresh Function 2).
[10 Dec 2008 19:28] Valeriy Kravchuk
Thank you for a problem report. Please, try to repeat with a newer version, 6.0.8, and inform about the results.
[11 Dec 2008 20:25] Akiko Marti
Tried 6.0.8-alpha as suggested, but it still crashes.

mysql> delete lineitem from lineitem inner join tmp_orderkey1 on l_orderkey=tmp_orderkey1.orderkey;
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> 

err file:

/usr/local/mysql608/libexec/mysqld'my_print_stacktrace+0x21 [0xb34d81]
/lib/amd64/libc.so.1'call_user_handler+0x2a7 [0xfffffd7ffed3d4df]
/usr/local/mysql608/libexec/mysqld'sel_node_create+0x8 [0xaa16a8]
[0x0]
[0x0]
081211 12:12:53 - mysqld got signal 11 ;
[2 Mar 2009 18:23] Valeriy Kravchuk
Please, try to repeat with a newer version, 6.0.9. In case of the same problem, please, send the results of:

explain select * from lineitem inner join tmp_orderkey1 on
l_orderkey=tmp_orderkey1.orderkey\G
[27 Mar 2009 21:10] Akiko Marti
I tried 6.0.10-alpha and the server crashed just like before when the same 'delete' was executed as shown below. I also found that if I create the tables with Myisam instead of InnoDB, then this delete statement just works fine (i.e. no crash).
So, the summary so far is:

delete lineitem from lineitem inner join tmp_orderkey1 on
l_orderkey=tmp_orderkey1.orderkey;

      Version   InnoDB  MyISAM

       5.1.29   OK      OK
       6.0.7    Crash
       6.0.8    Crash
       6.0.10   Crash   OK

Explain plan from MyISAM and InnoDB are pretty much the same (see the bottom of this file). The diffrence is a row count (1524 InnoDB vs. 1500 MyISAM).

err file on InnoDB crash:

090327 13:21:57 [Note] /usr/local/mysql.6010/libexec/mysqld: ready for connections.
Version: '6.0.10-alpha'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
/usr/local/mysql.6010/libexec/mysqld'my_print_stacktrace+0x21 [0xb374e1]
/lib/amd64/libc.so.1'call_user_handler+0x2a7 [0xfffffd7fff23dedf]
/usr/local/mysql.6010/libexec/mysqld'row_sel_field_store_in_mysql_format+0x2a0 [0xaa9c40]
090327 14:03:29 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=14
max_threads=100
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6575640 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x31e5bc2a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = fffffd7ffe909f88 thread_stack 0x40000
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 31e76c2b8 = delete lineitem from lineitem inner join tmp_orderkey1 on l_orderkey=tmp_orderkey1.orderkey
thd->thread_id=135
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
090327 14:03:38 mysqld_safe mysqld restarted
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
090327 14:03:45  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
090327 14:03:47  InnoDB: Started; log sequence number 1 2007516639
090327 14:03:47 [Note] /usr/local/mysql.6010/libexec/mysqld: ready for connections.
Version: '6.0.10-alpha'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution

Myisam explain plan:

mysql> explain select * from lineitem inner join tmp_orderkey1 on l_orderkey=tmp_orderkey1.orderkey\G

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: tmp_orderkey1
         type: index
possible_keys: i_tmp_okey
          key: i_tmp_okey
      key_len: 6
          ref: NULL
         rows: 1500
        Extra: Using index
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: lineitem
         type: ref
possible_keys: PRIMARY,i_l_orderkey,i_l_orderkey_quantity
          key: PRIMARY
      key_len: 4
          ref: dbt3.tmp_orderkey1.orderkey
         rows: 4
        Extra: Using index condition
2 rows in set (0.01 sec)

InnoDB explain plan:

mysql> explain select * from lineitem inner join tmp_orderkey1 on l_orderkey=tmp_orderkey1.orderkey\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: tmp_orderkey1
         type: index
possible_keys: i_tmp_okey
          key: i_tmp_okey
      key_len: 6
          ref: NULL
         rows: 1524
        Extra: Using index
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: lineitem
         type: ref
possible_keys: PRIMARY,i_l_orderkey,i_l_orderkey_quantity
          key: i_l_orderkey
      key_len: 4
          ref: dbt3.tmp_orderkey1.orderkey
         rows: 1
        Extra: Using index condition
2 rows in set (0.00 sec)
[27 Mar 2009 22:05] MySQL Verification Team
Thank you for the feedback. Looks like related to the bug: http://bugs.mysql.com/bug.php?id=37415. Could you please verify if you get similar backtrace?. Thanks in advance.
[31 Mar 2009 19:31] Akiko Marti
My crash is also from row_sel_field_store_in_mysql_format but the following is the all the info I get:

stack_bottom = fffffd7fff06df48 thread_stack 0x40000
/usr/local/mysql.6010/libexec/mysqld'my_print_stacktrace+0x21 [0x12a9969]
/lib/amd64/libc.so.1'call_user_handler+0x2a7 [0xfffffd7fff23dedf]
/usr/local/mysql.6010/libexec/mysqld'row_sel_field_store_in_mysql_format+0x2a0 [0x10458b0]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 33795c258 = delete lineitem from lineitem inner join tmp_orderkey1 on l_orderkey=tmp_orderkey1.orderkey
[14 Aug 2009 10:13] Sveta Smirnova
Thank you for the feedback.

Bug #37415 seems to be not repeatable now. Please test in your environment if problem which you meet is not repeatable too. Please use either version 6.0.11 or https://code.launchpad.net/~mysql/mysql-server/mysql-next Better is mysql-next, because this tree inherits 6.0 branch.
[14 Sep 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".