Bug #17364 mysqld crashes with signal 11
Submitted: 14 Feb 2006 7:59 Modified: 24 Dec 2006 15:39
Reporter: Cyril Bouthors Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.0.18-Debian_7 OS:Linux (Debian GNU/Linux)
Assigned to: CPU Architecture:Any

[14 Feb 2006 7:59] Cyril Bouthors
Description:
My upgrade from 4.0 to 5.0 failed with many sig 11:

Feb 14 08:15:02 sqla1 mysqld[30921]: mysqld got signal 11;
Feb 14 08:15:02 sqla1 mysqld[30921]: This could be because you hit a bug. It is also possible that this binary
Feb 14 08:15:02 sqla1 mysqld[30921]: or one of the libraries it was linked against is corrupt, improperly built,
Feb 14 08:15:02 sqla1 mysqld[30921]: or misconfigured. This error can also be caused by malfunctioning hardware.
Feb 14 08:15:02 sqla1 mysqld[30921]: We will try our best to scrape up some info that will hopefully help diagnose
Feb 14 08:15:02 sqla1 mysqld[30921]: the problem, but since we have already crashed, something is definitely wrong
Feb 14 08:15:02 sqla1 mysqld[30921]: and this may fail.
Feb 14 08:15:02 sqla1 mysqld[30921]:
Feb 14 08:15:02 sqla1 mysqld[30921]: key_buffer_size=16777216
Feb 14 08:15:02 sqla1 mysqld[30921]: read_buffer_size=131072
Feb 14 08:15:02 sqla1 mysqld[30921]: max_used_connections=8
Feb 14 08:15:02 sqla1 mysqld[30921]: max_connections=512
Feb 14 08:15:02 sqla1 mysqld[30921]: threads_connected=4
Feb 14 08:15:02 sqla1 mysqld[30921]: It is possible that mysqld could use up to
Feb 14 08:15:02 sqla1 mysqld[30921]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1130492 K
Feb 14 08:15:02 sqla1 mysqld[30921]: bytes of memory
Feb 14 08:15:02 sqla1 mysqld[30921]: Hope that's ok; if not, decrease some variables in the equation.
Feb 14 08:15:02 sqla1 mysqld[30921]:
Feb 14 08:15:02 sqla1 mysqld[30921]: thd=0x5a2039a0
Feb 14 08:15:02 sqla1 mysqld[30921]: Attempting backtrace. You can use the following information to find out
Feb 14 08:15:02 sqla1 mysqld[30921]: where mysqld died. If you see no messages after this, something went
Feb 14 08:15:02 sqla1 mysqld[30921]: terribly wrong...
Feb 14 08:15:02 sqla1 mysqld[30921]: Cannot determine thread, fp=0xbe3fe548, backtrace may not be correct.
Feb 14 08:15:02 sqla1 mysqld[30921]: Stack range sanity check OK, backtrace follows:
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x818f41c
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x4005c040
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x8114252
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81c6d61
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81f1ff6
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81a56ea
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81a899c
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81a8fa6
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81aa128
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81aaa74
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x40054f4c
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x402ab98a
Feb 14 08:15:02 sqla1 mysqld[30921]: New value of fp=(nil) failed sanity check, terminating stack trace!
Feb 14 08:15:02 sqla1 mysqld[30921]: Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
Feb 14 08:15:02 sqla1 mysqld[30921]: stack trace is much more helpful in diagnosing the problem, so please do
Feb 14 08:15:02 sqla1 mysqld[30921]: resolve it
Feb 14 08:15:02 sqla1 mysqld[30921]: Trying to get some variables.
Feb 14 08:15:02 sqla1 mysqld[30921]: Some pointers may be invalid and cause the dump to abort...
Feb 14 08:15:02 sqla1 mysqld[30921]: thd->query at 0x90a0120 = INSERT DELAYED INTO nuke_blocked_pagetracker (last_page ,page_date ,id_tracker) VALUES ('/modules.php?name=News&file=article&sid=47', '1139901318', '5394')
Feb 14 08:15:02 sqla1 mysqld[30921]: thd->thread_id=2544
Feb 14 08:15:02 sqla1 mysqld[30921]: The manual page at http://www.mysql.com/doc/en/Crashing.html contains
Feb 14 08:15:02 sqla1 mysqld[30921]: information that should help you find out what is causing the crash.
Feb 14 08:15:01 sqla1 mysqld[30921]: 060214  8:15:01 [ERROR] /usr/sbin/mysqld: Incorrect key file for table './gscv3/cron.MYI'; try to repair it
Feb 14 08:15:02 sqla1 mysqld[30921]: mysqld got signal 11;
Feb 14 08:15:02 sqla1 mysqld[30921]: This could be because you hit a bug. It is also possible that this binary
Feb 14 08:15:02 sqla1 mysqld[30921]: or one of the libraries it was linked against is corrupt, improperly built,
Feb 14 08:15:02 sqla1 mysqld[30921]: or misconfigured. This error can also be caused by malfunctioning hardware.
Feb 14 08:15:02 sqla1 mysqld[30921]: We will try our best to scrape up some info that will hopefully help diagnose
Feb 14 08:15:02 sqla1 mysqld[30921]: the problem, but since we have already crashed, something is definitely wrong
Feb 14 08:15:02 sqla1 mysqld[30921]: and this may fail.
Feb 14 08:15:02 sqla1 mysqld[30921]:
Feb 14 08:15:02 sqla1 mysqld[30921]: key_buffer_size=16777216
Feb 14 08:15:02 sqla1 mysqld[30921]: read_buffer_size=131072
Feb 14 08:15:02 sqla1 mysqld[30921]: max_used_connections=8
Feb 14 08:15:02 sqla1 mysqld[30921]: max_connections=512
Feb 14 08:15:02 sqla1 mysqld[30921]: threads_connected=4
Feb 14 08:15:02 sqla1 mysqld[30921]: It is possible that mysqld could use up to
Feb 14 08:15:02 sqla1 mysqld[30921]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1130492 K
Feb 14 08:15:02 sqla1 mysqld[30921]: bytes of memory
Feb 14 08:15:02 sqla1 mysqld[30921]: Hope that's ok; if not, decrease some variables in the equation.
Feb 14 08:15:02 sqla1 mysqld[30921]:
Feb 14 08:15:02 sqla1 mysqld[30921]: thd=0x5a2039a0
Feb 14 08:15:02 sqla1 mysqld[30921]: Attempting backtrace. You can use the following information to find out
Feb 14 08:15:02 sqla1 mysqld[30921]: where mysqld died. If you see no messages after this, something went
Feb 14 08:15:02 sqla1 mysqld[30921]: terribly wrong...
Feb 14 08:15:02 sqla1 mysqld[30921]: Cannot determine thread, fp=0xbe3fe548, backtrace may not be correct.
Feb 14 08:15:02 sqla1 mysqld[30921]: Stack range sanity check OK, backtrace follows:
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x818f41c
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x4005c040
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x8114252
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81c6d61
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81f1ff6
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81a56ea
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81a899c
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81a8fa6
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81aa128
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x81aaa74
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x40054f4c
Feb 14 08:15:02 sqla1 mysqld[30921]: 0x402ab98a
Feb 14 08:15:02 sqla1 mysqld[30921]: New value of fp=(nil) failed sanity check, terminating stack trace!
Feb 14 08:15:02 sqla1 mysqld[30921]: Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
Feb 14 08:15:02 sqla1 mysqld[30921]: stack trace is much more helpful in diagnosing the problem, so please do
Feb 14 08:15:02 sqla1 mysqld[30921]: resolve it
Feb 14 08:15:02 sqla1 mysqld[30921]: Trying to get some variables.
Feb 14 08:15:02 sqla1 mysqld[30921]: Some pointers may be invalid and cause the dump to abort...
Feb 14 08:15:02 sqla1 mysqld[30921]: thd->query at 0x90a0120 = INSERT DELAYED INTO nuke_blocked_pagetracker (last_page ,page_date ,id_tracker) VALUES ('/modules.php?name=News&file=article&sid=47', '1139901318', '5394')
Feb 14 08:15:02 sqla1 mysqld[30921]: thd->thread_id=2544
Feb 14 08:15:02 sqla1 mysqld[30921]: The manual page at http://www.mysql.com/doc/en/Crashing.html contains
Feb 14 08:15:02 sqla1 mysqld[30921]: information that should help you find out what is causing the crash.
Feb 14 08:15:02 sqla1 mysqld_safe[2486]: Number of processes running now: 0
Feb 14 08:15:02 sqla1 mysqld_safe[2488]: restarted
Feb 14 08:15:02 sqla1 mysqld_safe[2486]: Number of processes running now: 0
Feb 14 08:15:02 sqla1 mysqld_safe[2488]: restarted

root@sqla1:~# grep -c 'mysqld got signal 11' /var/log/daemon.log
12

I was forced to downgrade to 4.0 and restore the databases from the backups since they have been corrupted by the 12 crashes.

How to repeat:
To reproduce the bug, I just have to start the server, wait for my ~1k users to connect to my ~4k databases and mysqld keeps on crashing and crashing with signal 11.
[14 Feb 2006 9:06] Valeriy Kravchuk
Thank you for a (known!) problem report. How did you upgrade? Just installed new binaries over old data? 

Everybody who does like that and then perform INSERT DELAYED (as you did with nuke_blocked_pagetracker table, surely having varchar columns...) ends up just like you described. Please, search our bugs database for similar reports.

The only guaranteed (and documented) way to get proper results is to dump data in 4.0 and restore in 5.0.
[14 Feb 2006 11:34] Cyril Bouthors
This is indeed a problem with binary data but I expect the server to _explicitely_ complain about that, not segfault.
[14 Feb 2006 11:55] Valeriy Kravchuk
Looks similar to bug #13707 (that should be already fixed). Please, send the SHOW CREATE TABLE results for the table in that INSERT DELAYED statement.
[15 Mar 2006 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[24 Oct 2006 23:33] Christian Hammers
Hello

A Debian user reported the same error with 5.0.24 which is long after the fix from 5.0.16 (the see other bug comment).

bye,

-christian-

http://bugs.debian.org/386337   by   Christoph Hohmann <reboot@gmx.ch>

mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=1
max_connections=100
threads_connected=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 233983 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x8b8c980
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x4746accc, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81b98fc handle_segfault + 668
0xffffe420 _end + -141347824
(nil)
0x821ea07 _Z12mysql_insertP3THDP13st_table_listR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb + 1847
0x81d06b4 _Z21mysql_execute_commandP3THD + 10068
0x81d5009 _Z11mysql_parseP3THDPcj + 329
0x81d5560 _Z16dispatch_command19enum_server_commandP3THDPcj + 1120
0x81d67f8 _Z10do_commandP3THD + 136
0x81d7204 handle_one_connection + 2308
0x40054e60 _end + 932748880
0x402888ee _end + 935057630
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8c05110 = INSERT DELAYED INTO log SET ip="10.0.1.1", agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6 (Debian-1.5.dfsg+1.5.0.6-3)", request="/commsy/commsy.php?", method="GET", uid="", ulogin="guest", cid="192261", module="home", fct="index", param="", iid="-1"
thd->thread_id=8
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
------->8-------

If I can give any more information please ask.

-- System Information:
Debian Release: testing/unstable
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.5
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)

Versions of packages mysql-server-5.0 depends on:
ii  adduser                       3.85       Add and remove users and groups
ii  debconf [debconf-2.0]         1.4.72     Debian configuration management sy
ii  libc6                         2.3.6-7    GNU C Library: Shared libraries
ii  libdbi-perl                   1.50-2     Perl5 database interface by Tim Bu
ii  libgcc1                       1:4.1.1-13 GCC support library
ii  libmysqlclient15off           5.0.24a-3  mysql database client library
ii  libncurses5                   5.5-1      Shared libraries for terminal hand
ii  libreadline5                  5.1-7      GNU readline and history libraries
ii  libstdc++6                    4.1.1-13   The GNU Standard C++ Library v3
ii  libwrap0                      7.6.dbs-9  Wietse Venema's TCP wrappers libra
ii  lsb-base                      3.1-4      Linux Standard Base 3.1 init scrip
ii  mysql-client-5.0              5.0.24a-3  mysql database client binaries
ii  mysql-common                  5.0.24a-3  mysql database common files (e.g. 
ii  passwd                        1:4.0.13-7 change and administer password and
ii  perl                          5.8.8-6.1  Larry Wall's Practical Extraction 
ii  psmisc                        22.2-1     Utilities that use the proc filesy
ii  zlib1g                        1:1.2.3-11 compression library - runtime

 CREATE TABLE `log` (
  `id` int(11) NOT NULL auto_increment,
  `ip` varchar(15) default NULL,
  `agent` varchar(250) default NULL,
  `timestamp` timestamp NOT NULL default CURRENT_TIMESTAMP on update
CURRENT_TIMESTAMP,
  `request` varchar(250) default NULL,
  `method` varchar(10) default NULL,
  `uid` int(11) default NULL,
  `ulogin` varchar(250) default NULL,
  `cid` int(11) default NULL,
  `module` varchar(250) default NULL,
  `fct` varchar(250) default NULL,
  `param` varchar(250) default NULL,
  `iid` int(11) default NULL,
  PRIMARY KEY  (`id`),
  KEY `rid` (`cid`),
  KEY `timestamp` (`timestamp`)
) ENGINE=MyISAM AUTO_INCREMENT=4833261 DEFAULT CHARSET=latin1
[24 Nov 2006 15:39] Valeriy Kravchuk
Have you (or that customer) tried to run:

CHECK TABLE tbl_name FOR UPGRADE

on that table?
[24 Nov 2006 15:45] Christian Hammers
mysql_upgrade_shell is run on every server start, that should be sufficient, or?
But anyway, MySQL should print an error but not segfault...

bye,

-christian-
[25 Dec 2006 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".