MySQL Bugs: #42109: MySQL experienses bus error during shutdown

Bug #42109	MySQL experienses bus error during shutdown
Submitted:	14 Jan 2009 12:41	Modified:	27 Feb 2009 19:19
Reporter:	Lars-Erik BjÃ¸rk	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: Falcon storage engine	Severity:	S3 (Non-critical)
Version:	falcon-team, 6.0.10-bzr	OS:	Linux (Ubuntu 8.10)
Assigned to:	Christopher Powers	CPU Architecture:	Any
Tags:	F_SHUTDOWN, regression

Description:
If we create a table in Falcon, add an index, disconnect and shut down the server. The server gets a bus error during shutdown. This does not happen with innodb.

I have a callstack (produced on Solaris). Since Falcon is not mentioned in the stack, this may very well be a server problem.

[1] __lwp_kill(0x1, 0xb, 0xffffffff8f623160, 0x0, 0x0, 0x0), at 0xfffffd7ffedc42ea
  [2] _thr_kill(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffedbf2d3
=>[3] my_write_core(sig = 11), line 309 in "stacktrace.c"
  [4] handle_segfault(sig = 11), line 2672 in "mysqld.cc"
  [5] __sighndlr(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffedc1216
  [6] call_user_handler(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffedb64a2
  [7] sigacthandler(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7ffedb6688
  ---- called from signal handler with signal 11 (SIGSEGV) ------
  [8] my_hash_delete(hash = 0xeeeeeeeeeeeeeeee, record = 0x28b7640 "^D"), line 473 in "hash.c"
  [9] remove_from_used_mutex(locked_mutex = 0x7d46218, mutex = 0x28b7640), line 727 in "thr_mutex.c"
  [10] my_hash_iterate(hash = 0x2951550, action = 0x17fa3b0 = &`mysqld`thr_mutex.tmp.khepri20.27984.i`remove_from_used_mutex(safe_mutex_deadlock_t *locked_mutex, safe_mutex_t *mutex), argument = 0x28b7640), line 701 in "hash.c"
  [11] safe_mutex_free_deadlock_data(mp = 0x28b7640), line 592 in "thr_mutex.c"
  [12] safe_mutex_destroy(mp = 0x28b7640, file = 0x1923ed3 "mysqld.cc", line = 1467U), line 536 in "thr_mutex.c"
  [13] clean_up_mutexes(), line 1467 in "mysqld.cc"
  [14] mysqld_exit(exit_code = 0), line 1299 in "mysqld.cc"
  [15] main(argc = 6, argv = 0xfffffd7fffdfe658), line 4720 in "mysqld.cc"

How to repeat:
mysql> create table t1 (tall int) engine=falcon;
Query OK, 0 rows affected (0.03 sec)

mysql> create index i on t1 (tall);
Query OK, 0 rows affected (0.02 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> exit
Bye

$ ./client/mysqladmin -u root shutdown
090114 13:30:36 [Note] ./mysqld: Normal shutdown

090114 13:30:36 [Note] Event Scheduler: Purging the queue. 0 events
090114 13:30:36  InnoDB: Starting shutdown...
090114 13:30:37  InnoDB: Shutdown completed; log sequence number 0 46409
090114 13:30:38 [Note] ./mysqld: Shutdown complete

[1]+  Bus error               ./mysqld  (wd: ~/mysql/clean/sql)

(I have not started the sever with mtr when doing this, but with mysql_install_db and mysqld)

This issue seems to be related to ONLINE ADD INDEX, somehow. When creating the index OFFLINE, the crash does not occur:

mysql> CREATE TABLE t1(tall INT) Engine = Falcon;
Query OK, 0 rows affected (0.02 sec)

mysql> CREATE OFFLINE INDEX i ON t1(tall);
Query OK, 0 rows affected (0.06 sec)
Records: 0  Duplicates: 0  Warnings: 0

(exit, then shutdown, log is clean)

090114 14:14:45 [Note] ./sql/mysqld: ready for connections.
Version: '6.0.10-alpha-debug'  socket: '/tmp/mysql.sock'  port: 10010  Source distribution
090114 14:15:45 [Note] ./sql/mysqld: Normal shutdown

090114 14:15:45 [Note] Event Scheduler: Purging the queue. 0 events
090114 14:15:45  InnoDB: Starting shutdown...
090114 14:15:46  InnoDB: Shutdown completed; log sequence number 0 46409
090114 14:15:47 [Note] ./sql/mysqld: Shutdown complete

-----

In the case of default (ONLINE) CREATE INDEX, I see the following server output during shutdown with --falcon-debug-mask=65535:

090114 14:17:31  InnoDB: Starting shutdown...
090114 14:17:32  InnoDB: Shutdown completed; log sequence number 0 46409
30: Falcon shutdown
30: Cache flush: 33 pages, 16 writes in 0 seconds (33 pps)
090114 14:17:32 [Note] ./sql/mysqld: Shutdown complete

090114 14:17:32 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=10344644715844964239
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10102192105647527 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Verified just as described with 6.0.10 built recently from main bzr tree on Linux.

I've got no crashes from 6.0.8 on Windows, so this looks like a regression of a kind.

Chris wrote;  I have seen errors in the debugger here during mysqld shutdown, but only indicated through the Eclipse IDE--no call stacks. I'll see what I can find.

Unable to reproduce on mysql-6.0-falcon-team as of 2009-02-26.

Possibly corrected by shutdown fix in Bug#43048 "Falcon may crash during shutdown when running it embedded".