MySQL Bugs: #3919: mysqld crashes after update to 4.0.20

Bug #3919	mysqld crashes after update to 4.0.20
Submitted:	27 May 2004 18:11	Modified:	28 May 2004 10:30
Reporter:	Dimitrij HIlt	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	4.0.20-standard-log	OS:	Linux (Linux x86)
Assigned to:	Marko Mäkelä	CPU Architecture:	Any

Description:
After update from 4.0.18-standard-log to 4.0.20-standard-log ( binaries from www.mysql.com ) mysqld crashes after few seconds witch signal 11. Logfile says:

Version: '4.0.20-standard-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306
040527 14:22:52 Slave I/O thread: connected to master 'repl@mymaster:3306', replica
tion started in log 'binlog.4946' at position 7561983
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=0
max_connections=240
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 530430 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfeff6c8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80720d4
0x8250d48
0x8272e9e
0x80f87c2
0x80f8ce0
0x824e4fc
0x828452a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://www.mysql.com/doc/en/Using_stack_trace.html and follow instructions on how
to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Do you need any another information?

How to repeat:
Upgrade from 4.0.18 to 4.0.20, but not all mashines have this Problem.

Not enough information was provided for us to be able
to handle this bug. Please re-read the instructions at
http://bugs.mysql.com/how-to-report.php

If you can provide more information, feel free to add it
to this bug and change the status back to 'Open'.

Thank you for your interest in MySQL.

Additional info:

what is the query that causes a crash ?

Hi,

it seems to be same bug such as 3596. I got same backtrace:

0x80720d4 handle_segfault + 420
0x8250d48 pthread_sighandler + 184
0x8272e9e ftell + 14
0x80f87c2 srv_printf_innodb_monitor + 818
0x80f8ce0 srv_lock_timeout_and_monitor_thread + 512
0x824e4fc pthread_start_thread + 220
0x828452a thread_start + 4

Regards,

Dimitrij Hilt

Additional info:

mysqld die without any query

Dimitrij Hilt

Hi!

I guess the call of tmpfile() fails in dict0dict.c.

Workaround: check that the directory /tmp exists in the computer, and check that the disk is not full, and that the user running mysqld has full access rights to that directory, so that mysqld can create a temporary file there.

Marko should add diagnostics to:

heikki@hundin:~/mysql-4.0prod/innobase> grep -n tmpfile ./*/*.c
./dict/dict0dict.c:645: dict_foreign_err_file = tmpfile();
./lock/lock0lock.c:513: lock_latest_err_file = tmpfile();

or even better, make a new function os_file_create_tmpfile() and put diagnostics there.

Thank you for the bug report!

Heikki

TMPFILE(3)          Linux Programmer's Manual          TMPFILE(3)

NAME
       tmpfile - create a temporary file

SYNOPSIS
       #include <stdio.h>

       FILE *tmpfile (void);

DESCRIPTION
       The  tmpfile() function generates a unique temporary file
       name.   The  temporary  file  is  then  opened  in  binary
       read/write  (w+b)  mode.   The  file will be automatically
       deleted when it is closed or the program  terminates  nor
       mally.

RETURN VALUE
       The  tmpfile()  function  returns  a stream descriptor, or
       NULL if a unique  filename  cannot  be  generated  or  the
       unique file cannot be opened. In the latter case, errno is
       set to indicate the error.

ERRORS
       EACCES Search permission denied for  directory  in  file's
              path prefix.

       EEXIST Unable to generate a unique filename.

       EMFILE Too many file descriptors in use by the process.

       ENFILE Too many files open in the system.

       ENOSPC There  was  no room in the directory to add the new
              filename.

       EROFS  Read-only filesystem.

       EINTR  The call was interrupted by a signal.

       CONFORMING TO
              SVID 3, POSIX, BSD 4.3, ISO 9899, SUSv2

NOTES
       An error message may be written to stdout  if  the  stream
       cannot be opened.

       The standard does not specify the directory that tmpfile()
       will use. Glibc will try the path prefix P_tmpdir  defined
       in <stdio.h>, and if that fails the directory /tmp.

SEE ALSO
       exit(3), mktemp(3), mkstemp(3), tmpnam(3), tempnam(3)

Hi!

By the way, Marko should add diagnostics to all the cases where writing to those tmpfiles, or to the file innodb.status.xxxx, fails because the disk is full or for some other reason.

Maybe we should at the mysqld startup create a fixed-size innodb.status.xxxxx and fixed-size innodb_foreign_key_err and innodb_deadlock_err files, so that it would be less probable that writing will fail because of a full disk.

Regards,

Heikki

Hi!

Actually, maybe we should remove the call of tmpfile() and create those files to the datadir, like the innodb.status file? Then we do not need to assume anything about the mysqld access rights to /tmp.

Regards,

Heikki

Thanks to Heikki,

chmod 1777 /tmp resolv this problem. But 'signal 11'?!

Dimitrij Hilt

Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html

Additional info:

Like Heikki wrote, there were two unsafe calls to tmpfile(), invoked at server start-up.  I added a diagnostics message.  I don't think that it is a good idea to create these two files in datadir, because on some systems, /tmp resides in virtual memory (such as Linux shm fs) and may thus be faster than datadir.