MySQL Bugs: #41013: main.bootstrap coredumps in 6.0-rpl

Bug #41013	main.bootstrap coredumps in 6.0-rpl
Submitted:	25 Nov 2008 13:53	Modified:	12 Nov 2009 19:10
Reporter:	Sven Sandberg	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: General	Severity:	S2 (Serious)
Version:	6.0-rpl	OS:	Any
Assigned to:	Zhenxing He	CPU Architecture:	Any
Tags:	6.0-rpl-green, bootstrap, F_SERVER, pushbuild, test failure

Description:
pushbuild failure seems to happen deterministically on all pushes and hosts since 29 Oct in 6.0-rpl (and no other tree):

main.bootstrap                           [ fail ]

CURRENT_TEST: main.bootstrap
sh: line 1: 20080 Segmentation fault      (core dumped) /data0/pushbuild/pb/bzr_mysql-6.0-rpl/109/bzr_mysql-6.0-rpl-exp/sql/mysqld --no-defaults --bootstrap --basedir=/data0/pushbuild/pb/bzr_mysql-6.0-rpl/109/bzr_mysql-6.0-rpl-exp --datadir=/dev/shm/var-embedded-102/install.db --loose-skip-innodb --loose-skip-ndbcluster --tmpdir=/dev/shm/var-embedded-102/tmp/ --core-file --language=/data0/pushbuild/pb/bzr_mysql-6.0-rpl/109/bzr_mysql-6.0-rpl-exp/sql/share/english --character-sets-dir=/data0/pushbuild/pb/bzr_mysql-6.0-rpl/109/bzr_mysql-6.0-rpl-exp/sql/share/charsets --datadir=/dev/shm/var-embedded-102/mysqld.1/data/ </dev/shm/var-embedded-102/tmp/bootstrap_test.sql >>/dev/shm/var-embedded-102/tmp/bootstrap.log 2>&1
mysqltest: At line 20: command "$MYSQLD_BOOTSTRAP_CMD < $MYSQLTEST_VARDIR/tmp/bootstrap_test.sql >> $MYSQLTEST_VARDIR/tmp/bootstrap.log 2>&1" failed

The result from queries just before the failure was:
drop table if exists t1;

 - saving '/dev/shm/var-embedded-102/log/main.bootstrap/' to '/dev/shm/var-embedded-102/log/main.bootstrap/'
 - found 'core.20080' (0/5)

Trying 'gdb' to get a backtrace
Core generated by '/data0/pushbuild/pb/bzr_mysql-6.0-rpl/109/bzr_mysql-6.0-rpl-exp/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/data0/pushbuild/pb/bzr_mysql-6.0-rpl/109/bzr_mysql-6.0-rpl-exp/sql/mysqld --no'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002b5cae5b44c5 in pthread_kill () from /lib64/libpthread.so.0
#0  0x00002b5cae5b44c5 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000d727e1 in my_write_core (sig=11) at stacktrace.c:307
#2  0x0000000000708a49 in handle_segfault (sig=11) at mysqld.cc:2670
#3  <signal handler called>
#4  0x00000000009105f4 in intern_plugin_lock (lex=0x1e5a0b8, rc=0x13b1480, 
    sFile=0xf24c78 "handler.cc", uLine=169) at sql_plugin.cc:621
#5  0x0000000000910afa in plugin_lock (thd=0x1e584a8, ptr=0x40880b28, 
    sFile=0xf24c78 "handler.cc", uLine=169) at sql_plugin.cc:659
#6  0x0000000000859049 in ha_lock_engine (thd=0x1e584a8, hton=0x1e4dc88)
    at handler.cc:169
#7  0x0000000000859667 in ha_resolve_by_legacy_type (thd=0x1e584a8, 
    db_type=DB_TYPE_MYISAM) at handler.cc:192
#8  0x00000000008596c5 in ha_checktype (thd=0x1e584a8, 
    database_type=DB_TYPE_MYISAM, no_substitute=false, report_error=true)
    at handler.cc:208
#9  0x000000000086e2f4 in check_engine (thd=0x1e584a8, 
    table_name=0x1e6b1d0 "t1", create_info=0x408812b0) at sql_table.cc:7690
#10 0x00000000008782eb in mysql_create_table_no_lock (thd=0x1e584a8, 
    db=0x1e6b570 "test", table_name=0x1e6b1d0 "t1", create_info=0x408812b0, 
    alter_info=0x40881be0, internal_tmp_table=false, select_field_count=0)
    at sql_table.cc:3320
#11 0x0000000000879900 in mysql_create_table (thd=0x1e584a8, 
    db=0x1e6b570 "test", table_name=0x1e6b1d0 "t1", create_info=0x408812b0, 
    alter_info=0x40881be0, internal_tmp_table=false, select_field_count=0)
    at sql_table.cc:3793
#12 0x000000000071b72c in mysql_execute_command (thd=0x1e584a8)
    at sql_parse.cc:2589
#13 0x000000000072245b in mysql_parse (thd=0x1e584a8, 
    inBuf=0x1e6b130 "CREATE TABLE t1(a int)", length=22, 
    found_semicolon=0x40882178) at sql_parse.cc:5726
#14 0x0000000000724f6a in handle_bootstrap (arg=0x1e584a8) at sql_parse.cc:491
#15 0x00002b5cae5b0193 in start_thread () from /lib64/libpthread.so.0
#16 0x00002b5caebe447d in clone () from /lib64/libc.so.6
#17 0x0000000000000000 in ?? ()

Thread 3 (process 20080):
#0  0x00002b5cae5b37f6 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000000000d782a4 in safe_cond_wait (cond=0x13a96e0, mp=0x13a8da0, 
    file=0xea2d58 "mysqld.cc", line=4955) at thr_mutex.c:249
#2  0x0000000000705643 in bootstrap (file=0x2b5caed4f6a0) at mysqld.cc:4955
#3  0x000000000070c4a2 in main (argc=12, argv=0x7ffffc836f98)
    at mysqld.cc:4652

Thread 2 (process 25402):
#0  0x00002b5cae5b6ef8 in do_sigwait () from /lib64/libpthread.so.0
#1  0x00002b5cae5b6f9d in sigwait () from /lib64/libpthread.so.0
#2  0x0000000000708259 in signal_hand (arg=0x0) at mysqld.cc:2873
#3  0x00002b5cae5b0193 in start_thread () from /lib64/libpthread.so.0
#4  0x00002b5caebe447d in clone () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 1 (process 25403):
#0  0x00002b5cae5b44c5 in pthread_kill () from /lib64/libpthread.so.0
#1  0x0000000000d727e1 in my_write_core (sig=11) at stacktrace.c:307
#2  0x0000000000708a49 in handle_segfault (sig=11) at mysqld.cc:2670
#3  <signal handler called>
#4  0x00000000009105f4 in intern_plugin_lock (lex=0x1e5a0b8, rc=0x13b1480, 
    sFile=0xf24c78 "handler.cc", uLine=169) at sql_plugin.cc:621
#5  0x0000000000910afa in plugin_lock (thd=0x1e584a8, ptr=0x40880b28, 
    sFile=0xf24c78 "handler.cc", uLine=169) at sql_plugin.cc:659
#6  0x0000000000859049 in ha_lock_engine (thd=0x1e584a8, hton=0x1e4dc88)
    at handler.cc:169
#7  0x0000000000859667 in ha_resolve_by_legacy_type (thd=0x1e584a8, 
    db_type=DB_TYPE_MYISAM) at handler.cc:192
#8  0x00000000008596c5 in ha_checktype (thd=0x1e584a8, 
    database_type=DB_TYPE_MYISAM, no_substitute=false, report_error=true)
    at handler.cc:208
#9  0x000000000086e2f4 in check_engine (thd=0x1e584a8, 
    table_name=0x1e6b1d0 "t1", create_info=0x408812b0) at sql_table.cc:7690
#10 0x00000000008782eb in mysql_create_table_no_lock (thd=0x1e584a8, 
    db=0x1e6b570 "test", table_name=0x1e6b1d0 "t1", create_info=0x408812b0, 
    alter_info=0x40881be0, internal_tmp_table=false, select_field_count=0)
    at sql_table.cc:3320
#11 0x0000000000879900 in mysql_create_table (thd=0x1e584a8, 
    db=0x1e6b570 "test", table_name=0x1e6b1d0 "t1", create_info=0x408812b0, 
    alter_info=0x40881be0, internal_tmp_table=false, select_field_count=0)
    at sql_table.cc:3793
#12 0x000000000071b72c in mysql_execute_command (thd=0x1e584a8)
    at sql_parse.cc:2589
#13 0x000000000072245b in mysql_parse (thd=0x1e584a8, 
    inBuf=0x1e6b130 "CREATE TABLE t1(a int)", length=22, 
    found_semicolon=0x40882178) at sql_parse.cc:5726
#14 0x0000000000724f6a in handle_bootstrap (arg=0x1e584a8) at sql_parse.cc:491
#15 0x00002b5cae5b0193 in start_thread () from /lib64/libpthread.so.0
#16 0x00002b5caebe447d in clone () from /lib64/libc.so.6
#17 0x0000000000000000 in ?? ()

Retrying test, attempt(2/3)...

How to repeat:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0-rpl&order=109 debx86-b/embedded
xref: http://tinyurl.com/56c9ha

See also BUG#41154.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/60470

2741 He Zhenxing	2008-12-03
      BUG#41013 main.bootstrap coredumps in 6.0-rpl
      
      When a storage engine failed to initialize before allocated slot number,
      the slot number would be 0, and when later finalizing this plugin, it would
      accidentally unplug the storage engine currently uses slot 0.
      
      This patch fixed this problem by add a new macro value HA_SLOT_UNDEF to
      distinguish undefined slot number from slot 0.

Some left over state while executing main.bootstrap is causing failures in concurrent_insert_func

Attachment: main.concurrent_insert_func.output (application/octet-stream, text), 12.65 KiB.

Alfranio, the error in main.concurrent_insert_func is BUG#39862. It has happened even without previous failures in main.bootstrap, e.g., on https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-5.1-rpl&order=55 debx86-b/n_mix . So I think it is unrelated.

Pushed into 6.0.10-alpha (revid:luis.soares@sun.com-20090129165607-wiskabxm948yx463) (version source revid:luis.soares@sun.com-20090129163120-e2ntks4wgpqde6zt) (merge vers: 6.0.10-alpha) (pib:6)

Noted in 6.0.10 changelog.

When a storage engine plugin failed to initialize before allocating a
slot number, it would acidentally unplug the engine in slot 0.

Bug: http://bugs.mysql.com/bug.php?id=42967 marked as duplicate of this one.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/85470

3116 He Zhenxing	2009-10-02
      Backport BUG#41013 main.bootstrap coredumps in 6.0-rpl
      
      When a storage engine failed to initialize before allocated slot number,
      the slot number would be 0, and when later finalizing this plugin, it would
      accidentally unplug the storage engine currently uses slot 0.
      
      This patch fixed this problem by add a new macro value HA_SLOT_UNDEF to
      distinguish undefined slot number from slot 0.

Pushed into 6.0.14-alpha (revid:alik@sun.com-20091027094604-9p7kplu1vd2cvcju) (version source revid:zhenxing.he@sun.com-20091026140226-uhnqejkyqx1aeilc) (merge vers: 6.0.14-alpha) (pib:13)

Already fixed in 6.0.10.

Pushed into 5.5.0-beta (revid:alik@sun.com-20091110093229-0bh5hix780cyeicl) (version source revid:alik@sun.com-20091027095744-rf45u3x3q5d1f5y0) (merge vers: 5.5.0-beta) (pib:13)

Noted in 5.5.0 changelog.