MySQL Bugs: #25600: MySQL segfaults during boot

Bug #25600	MySQL segfaults during boot
Submitted:	12 Jan 2007 22:57	Modified:	16 Jan 2007 9:44
Reporter:	Michael Watters	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	4.1.21	OS:	Linux (CentOS 4.4)
Assigned to:	Heikki Tuuri	CPU Architecture:	Any
Tags:	innodb, segfault, xen

Description:
MySQL appears to segfault during the boot process however logging in and starting the service manually produces no errors.  Here is a copy of the stack dump that I resolved.

This is a CentOS 4.4 VM running on Xen 3.0.3

0x8133fc0 handle_segfault + 656
0x4990b7 (?)
0x497487 (?)
0x4942fe (?)
0x8370bbb create_temp_file + 267
0x81bb4b0 innobase_mysql_tmpfile + 80
0x832c753 os_file_create_tmpfile + 19
0x8205f0f innobase_start_or_create_for_mysql + 2799
0x81bbe6c _Z13innobase_initv + 988
0x81b31dd _Z7ha_initv + 29
0x8137675 main + 2149
0x1bd261 (?)
0x80c7471 _start + 33

How to repeat:
Shut down the VM, run "xm create -c domain.cfg"

Thank you for a problem report. Please, send the entire error log and my.cnf file content.

Hi!

Thus may be a bug in the Xen OS virtualization software. The stack trace looks like it seg faults in the operating system call that should create a temporary file.

Regards,

Heikki

Hi!

Do you have a directory /tmp in your computer? Maybe Xen is not able to handle the tmpfile() call if there is no /tmp?

Regards,

Heikki

man tmpfile:

TMPFILE(3)                 Linux Programmer's Manual                TMPFILE(3)

NAME
       tmpfile - create a temporary file

SYNOPSIS
       #include <stdio.h>

       FILE *tmpfile (void);

DESCRIPTION
       The tmpfile() function generates a unique temporary filename.  The tem-
       porary file is then opened in binary read/write (w+b) mode.   The  file
       will  be  automatically deleted when it is closed or the program termi-
       nates normally.

RETURN VALUE
       The tmpfile() function returns a stream descriptor, or NULL if a unique
       filename  cannot  be  generated or the unique file cannot be opened. In
       the latter case, errno is set to indicate the error.

ERRORS
       EACCES Search permission denied for directory in file's path prefix.

       EEXIST Unable to generate a unique filename.

       EINTR  The call was interrupted by a signal.

       EMFILE Too many file descriptors in use by the process.

       ENFILE Too many files open in the system.

       ENOSPC There was no room in the directory to add the new filename.

       EROFS  Read-only filesystem.

CONFORMING TO
       SVID 3, POSIX, BSD 4.3, ISO 9899, SUSv2

NOTES
       An error message may be written to  stdout  if  the  stream  cannot  be
       opened.

       The  standard  does  not specify the directory that tmpfile() will use.
       Glibc will try the path prefix P_tmpdir defined in  <stdio.h>,  and  if
       that fails the directory /tmp.

SEE ALSO
       exit(3), mkstemp(3), mktemp(3), tempnam(3), tmpnam(3)

I just upgrade to Xen 3.0.4 and it does the same thing.  Here is a copy of the error log.  I'm not sure why it can't create a file under /tmp when it's mounted and set to 777.

070115 17:18:45  mysqld started
^G/usr/sbin/mysqld: Can't create/write to file '/tmp/ib7bKKn9' (Errcode: 13)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 217599
 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)

Here's the fstab entry for tmp.

/usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0)

I believe I found the problem, MySQL is starting before tmpDSK gets mounted so it fails.  I unmounted /tmp and set the directory to 777 (was 755) and rebooted, MySQL starts fine now.  I'm not sure where tmpDSK gets mounted in the first place since it doesn't exist anywhere in the init scripts.

Also, creating a new temp directory for mysql under /home/temp allows it to start without a problem.

So, this is not a result of bug in MySQL code.

InnoDB creates a few temporary files for diagnostic messages (to avoid using sprintf() which could cause buffer overflows). If the files cannot be created when InnoDB starts, an assertion failure will occur.

For what it is worth, we rely on tmpfile() only on Netware. On other platforms, the files are created by the MySQL function create_temp_file().