MySQL Bugs: #20555: mysqld got signal 11;

Bug #20555	mysqld got signal 11;
Submitted:	20 Jun 2006 7:20	Modified:	21 Aug 2006 15:25
Reporter:	Juan Carlos Moreno	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	mysql-5.0.22-max (MySQL Community Editio	OS:	Linux (Linux Debian 3.1)
Assigned to:		CPU Architecture:	Any

Description:
mysql dies so often when using storage engine NDB. The datanodes are ok, but mysql dies and is restarted, so it disconnect from manager host, and reconnect.

The query is:

(select distinct param22 as sel from FLIGHT_GRID
 where param00 = '_1150747655317')
  union
(select distinct param23 as sel from FLIGHT_GRID
 where param00 = '_1150747655317')
  order by sel

That normaly returns:

mysql> (select distinct param22 as sel from FLIGHT_GRID
    ->  where param00 = '_1150747655317')
    ->   union
    -> (select distinct param23 as sel from FLIGHT_GRID
    ->  where param00 = '_1150747655317')
    ->   order by sel;
+------+
| sel  |
+------+
| NULL |
|    0 |
|    1 |
+------+
3 rows in set (0.01 sec)

but, same query some times at day:

060619 15:39:25  mysqld restarted
060619 15:39:29 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: '5.0.22-max-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306
MySQL Community Edition - Experimental (GPL)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose
the problem, but since we have already crashed, something is definitely
wrong
and this may fail.

key_buffer_size=134217728
read_buffer_size=4190208
max_used_connections=91
max_connections=1024
threads_connected=88
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections =
126968 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0xaafd8d80
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xacd46cfc, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8192742
0xffffe420
0xacd47094
0x8264b54
0x8264e10
0x8264c90
0x8264db4
0x82634c4
0x81a4572
0x81aaf0d
0x81a2b26
0x81a23c3
0x81a1904
0xb7fbeca3
0xb7ed8f5a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and
follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xaabed008  is invalid pointer
thd->thread_id=32520
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

resolve_stack_dump -s /tmp/mysqld.sym -n ./mysql.stack
0x8192742 handle_segfault + 430
0xffffe420 _end + -142019208
0xad9aa094 _end + -1524387348
0x8264b54 invalidate_table__11Query_cacheP17Query_cache_block + 68
0x8264e10 insert_table__11Query_cacheUiPcP23Query_cache_block_tableUiUcPFP3THDPcUiPUx_cUx + 84
0x8264c90 register_tables_from_list__11Query_cacheP13st_table_listUiP23Query_cache_block_table + 296
0x8264db4 register_all_tables__11Query_cacheP17Query_cache_blockP13st_table_listUi + 28
0x82634c4 store_query__11Query_cacheP3THDP13st_table_list + 652
0x81a4572 mysql_execute_command__FP3THD + 1310
0x81aaf0d mysql_parse__FP3THDPcUi + 281
0x81a2b26 dispatch_command__F19enum_server_commandP3THDPcUi + 1878
0x81a23c3 do_command__FP3THD + 195
0x81a1904 handle_one_connection + 764
0xb7f3eca3 _end + -1350763013
0xb7e58f5a _end + -1351704398

C compiler:    2.95.3
C++ compiler:  2.95.3
Environment: System: Linux XXXX 2.6.13 #3 SMP Fri Sep 9 19:48:32 CEST 2005 i686 GNU/Linux
Architecture: i686

Some paths:  /usr/bin/perl /usr/bin/make /usr/bin/gcc /usr/bin/cc
GCC: Leyendo especificaciones de /usr/lib/gcc-lib/i486-linux/3.3.5/specs
Configurado con: ../src/configure -v --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared --enable-__cxa_atexit --with-system-zlib --enable-nls
--without-included-gettext --enable-clocale=gnu --enable-debug --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc i486-linux
Modelo de hilos: posix
gcc versión 3.3.5 (Debian 1:3.3.5-13)
Compilation info: CC='ccache gcc'  CFLAGS='-O2 -mcpu=pentiumpro'  CXX='ccache gcc'  CXXFLAGS='-O2 -mcpu=pentiumpro -felide-constructors'
LDFLAGS='-L/usr/local/lib'  ASFLAGS=''
LIBC:
lrwxrwxrwx  1 root root 13 2006-06-17 00:43 /lib/libc.so.6 -> libc-2.3.2.so
-rwxr-xr-x  1 root root 1244752 2006-04-02 15:32 /lib/libc-2.3.2.so
-rw-r--r--  1 root root 2642038 2006-04-02 15:32 /usr/lib/libc.a
-rwxr-xr-x  1 root root 204 2006-04-02 15:22 /usr/lib/libc.so
Configure command: ./configure '--prefix=/usr/local/mysql' '--localstatedir=/usr/local/mysql/data' '--libexecdir=/usr/local/mysql/bin'
'--with-comment=MySQL Community Edition - Experimental (GPL)' '--with-server-suffix=-max' '--enable-thread-safe-client' '--enable-local-infile'
'--enable-assembler' '--with-pic' '--with-fast-mutexes' '--disable-shared' '--with-berkeley-db' '--with-big-tables' '--with-yassl' '--with-readline'
'--with-archive-storage-engine' '--with-blackhole-storage-engine' '--with-ndbcluster' '--with-csv-storage-engine' '--with-example-storage-engine'
'--with-federated-storage-engine' '--with-innodb' '--with-extra-charsets=all' 'CC=ccache gcc' 'CFLAGS=-O2 -mcpu=pentiumpro' 'CPPFLAGS=-I/usr/local/include
-DDEFAULT_THREAD_STACK=126976' 'CXXFLAGS=-O2 -mcpu=pentiumpro -felide-constructors' 'CXX=ccache gcc' 'LDFLAGS=-L/usr/local/lib'

How to repeat:
aleatory. three or four times at day, using the same query works or it crashes.

Suggested fix:
dunno

Thank you for a problem report. Please, send the SHOW CREATE TABLE and SHOW TABLE STATUS results for that FLIGHT_GRID table. Can you, please, try to disable query cache and check, if this problem will appear again. How much RAM do you have on your machine?

SHOW CREATE TABLE:

CREATE TABLE `FLIGHT_GRID` (
  `productid` int(11) NOT NULL auto_increment,
  `categoryid` decimal(1,0) NOT NULL default '0',
  `product` varchar(16) default NULL,
  `param00` varchar(32) default NULL,
  `param01` varchar(16) default NULL,
  `param02` decimal(12,0) default NULL,
  `param03` decimal(12,0) default NULL,
  `param04` date default NULL,
  `param05` date default NULL,
  `param06` varchar(32) default NULL,
  `param07` varchar(8) default NULL,
  `param08` decimal(12,0) default NULL,
  `param09` varchar(3) default NULL,
  `param10` varchar(3) default NULL,
  `param11` varchar(15) default NULL,
  `param14` date default NULL,
  `param15` date default NULL,
  `param16` varchar(32) default NULL,
  `param17` varchar(8) default '0',
  `param18` varchar(2) default '0',
  `param19` varchar(15) default NULL,
  `param20` varchar(2) default NULL,
  `param21` varchar(2) default NULL,
  `param22` decimal(2,0) default NULL,
  `param23` decimal(2,0) default NULL,
  `param24` varchar(8) default NULL,
  `param25` varchar(15) default NULL,
  `param26` varchar(100) default NULL,
  `param27` decimal(12,0) default NULL,
  `param30` varchar(16) default NULL,
  `param31` decimal(12,0) default NULL,
  `param32` decimal(12,0) default NULL,
  `param33` decimal(12,0) default NULL,
  `param34` decimal(12,0) default NULL,
  `param35` decimal(12,0) default NULL,
  `param36` decimal(12,0) default NULL,
  `param37` decimal(12,0) default '0',
  `param38` decimal(12,0) default '0',
  `param39` decimal(12,0) default NULL,
  `param40` decimal(12,0) NOT NULL default '0',
  `param41` varchar(6) default NULL,
  `param51` varchar(50) NOT NULL,
  `param52` time default NULL,
  `param53` time default NULL,
  `param54` time default NULL,
  `param55` time default NULL,
  `param56` varchar(5) default NULL,
  `param57` varchar(5) default NULL,
  `param58` char(1) default '0',
  `param59` char(1) default '0',
  `param98` decimal(1,0) NOT NULL default '0',
  `param99` int(11) default NULL,
  `booked` enum('Y','N') default 'N',
  `localizador` varchar(20) NOT NULL default ' ',
  `localizadorVuelta` varchar(20) NOT NULL,
  `SearchID` varchar(20) NOT NULL,
  `commission` decimal(12,0) default NULL,
  `commission_applied` decimal(12,0) default NULL,
  `discount` decimal(12,0) NOT NULL default '0',
  `fee_adt` decimal(12,0) NOT NULL default '0',
  `fee_chd` decimal(12,0) NOT NULL default '0',
  `fee_inf` decimal(12,0) NOT NULL default '0',
  `uatp` enum('Y','N') default 'Y',
  `wnh_datecreation` timestamp NOT NULL default CURRENT_TIMESTAMP,
  `fee_included` enum('Y','N') default NULL,
  `permission` enum('S','B','C','T') default 'S',
  `xml_descr` text,
  `xml_pricing` varchar(255) default NULL,
  PRIMARY KEY  (`productid`),
  KEY `product` (`product`),
  KEY `param00` (`param00`),
  KEY `categoryid` (`categoryid`),
  KEY `param40` (`param40`),
  KEY `param01` (`param01`),
  KEY `param06` (`param06`),
  KEY `param16` (`param16`),
  KEY `param52` (`param52`),
  KEY `param53` (`param53`),
  KEY `param54` (`param54`),
  KEY `param55` (`param55`),
  KEY `param22` (`param22`),
  KEY `param98` (`param98`),
  KEY `param08` (`param08`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1 

SHOW TABLE STATUS:

FLIGHT_GRID     ndbcluster      10      Dynamic 47268   1308    67764224        0       0       0       545141  NULL    NULL    NULL    latin1_swedish_ci    NULL             number_of_replicas: 2

cat /proc/meminfo
MemTotal:      8310592 kB
MemFree:        286492 kB
Buffers:        296748 kB
Cached:        5297204 kB
SwapCached:          0 kB
Active:        5116512 kB
Inactive:      2559164 kB
HighTotal:     7470528 kB
HighFree:       106888 kB
LowTotal:       840064 kB
LowFree:        179604 kB
SwapTotal:    13068868 kB
SwapFree:     13068868 kB
Dirty:            4296 kB
Writeback:           0 kB
Mapped:        2143000 kB
Slab:           319240 kB
CommitLimit:  17224164 kB
Committed_AS:  4474340 kB
PageTables:       9632 kB
VmallocTotal:   118776 kB
VmallocUsed:     24252 kB
VmallocChunk:    93440 kB

top sort by memory:
top - 11:59:32 up 22 days, 11:08,  3 users,  load average: 0.13, 0.06, 0.02
Tasks: 114 total,   1 running, 113 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0% us,  0.0% sy,  0.0% ni, 99.8% id,  0.2% wa,  0.0% hi,  0.0% si
Mem:   8310592k total,  8029020k used,   281572k free,   296752k buffers
Swap: 13068868k total,        0k used, 13068868k free,  5301348k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27203 root      15   0 2872m 1.3g 2608 S  0.0 17.0  12:48.55 ndbd
8182 root      18   0 1317m 336m  37m S  0.0  4.1  14:51.10 java
2220 mysql     16   0  589m 110m 4552 S  0.0  1.4   8:43.20 mysqld

I will disable query cache on mysql node 2 tomorrow, i'm doing some test now...

Thank you for the additional information. Please, reopen this report when you'll have results of testing without query cache.

If I disable query_cache, the problem dissapears.

do you need more information?

Best Regards
Juan Carlos Moreno
http://www.ackstorm.es

Please, send the results of:

EXPLAIN (select distinct param22 as sel from FLIGHT_GRID
 where param00 = '_1150747655317')
  union
(select distinct param23 as sel from FLIGHT_GRID
 where param00 = '_1150747655317')
  order by sel\G

Do you really have 

query_cache_size = 32M

as in default my-huge.cnf? Is it possible to get a crash with query chache on, but smaller key_buffer, for example? Maybe, you just have not enough RAM for all processes running.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".