MySQL Bugs: #21067: Server crash with Signal 11

Bug #21067	Server crash with Signal 11
Submitted:	14 Jul 2006 16:15	Modified:	30 Aug 2006 11:20
Reporter:	Bj?rn-Arne Jensen	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	5.0.21-standard-log	OS:	Linux (SLES9 SP3 for PPC)
Assigned to:		CPU Architecture:	Any
Tags:	crash, Memory, PPC, Signal11

Description:
mysqld crashed with Signal 11 

In the error_log appears the following error:

This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1073741824
read_buffer_size=2093056
max_used_connections=257
max_connections=256
threads_connected=38
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2096126 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Writing a core file

-----------------------------------------------------------------------------
The Machine has 32GB of Memory 2GB memory were free 
29GB were used by a previous crash of mysql and  was not released.
Mem:  32113108k total, 29685260k used,  2427848k free,  1306376k buffers
Swap:  8390648k total,        4k used,  8390644k free, 12709372k cached

Most of our tables were MyIsam Tables. 2 Tables are memory Tables
Server crashed while doing a load data infile via Replication

How to repeat:
It is repeatable every time. We just have to wait some hours.

Thank you for a problem report. Please, send also uname -a results and exact URL for the MySQL server package you used.

The URL was:
http://dev.mysql.com/get/Downloads/MySQL-5.0/mysql-standard-5.0.21-linux-powerpc-glibc23.t...

uname -a 
Linux sql01 2.6.5-7.244-pseries64 #1 SMP Mon Dec 12 18:32:25 UTC 2005 ppc64 ppc64 ppc64 GNU/Linux

I am just uploading another tar File. We installed mysql-5.0.22-debug Version.
In that tar File there are 2 core-dumps, which where written by crash.
We started mysql without the debug option, because it was too slow, and we had in one hour 15GB of trace File. 
Is there a possibility to limit the debug information? 

The output of gdb64 is:
sql01:/opt/mysql/var # gdb64 ../bin/mysqld core.17510     
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "ppc64-suse-linux"...Using host libthread_db library 
"/lib64/tls/libthread_db.so.1".

Core was generated by `/opt/mysql/bin/mysqld --basedir=/opt/mysql --datadir=/opt
/mysql/var --pid-file='.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/tls/librt.so.1...done.
Loaded symbols for /lib/tls/librt.so.1
Reading symbols from /lib/tls/libpthread.so.0...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
#0  0x0ffc0a0c in pthread_kill () from /lib/tls/libpthread.so.0
(gdb) backtrace
#0  0x0ffc0a0c in pthread_kill () from /lib/tls/libpthread.so.0
#1  0x103615dc in write_core (sig=Cannot access memory at address 0x13
) at stacktrace.c:220
#2  0x101cf2f8 in handle_segfault (sig=11) at mysqld.cc:2094
#3  <signal handler called>
Cannot access memory at address 0xfbebdb70

And in the other core-file:
sql01:/opt/mysql/var # gdb64 ../bin/mysqld core.13176 
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "ppc64-suse-linux"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".

Core was generated by `/opt/mysql/bin/mysqld --basedir=/opt/mysql --datadir=/opt/mysql/var --pid-file='.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/tls/librt.so.1...done.
Loaded symbols for /lib/tls/librt.so.1
Reading symbols from /lib/tls/libpthread.so.0...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
#0  0x0ffc0a0c in pthread_kill () from /lib/tls/libpthread.so.0
(gdb) backtrace
#0  0x0ffc0a0c in pthread_kill () from /lib/tls/libpthread.so.0
#1  0x103615dc in write_core (sig=Cannot access memory at address 0x13
) at stacktrace.c:220
#2  0x101cf2f8 in handle_segfault (sig=11) at mysqld.cc:2094
#3  <signal handler called>
Cannot access memory at address 0xffea0b70

Today i started mysqld with gdb64. Here is the output:

.....
[Thread 4266214624 (zombie) exited]
[Thread 4264531168 (zombie) exited]
[Thread 4252144864 (zombie) exited]
[Thread 4222350560 (zombie) exited]
[Thread 4287755488 (zombie) exited]
[Thread 4256212192 (zombie) exited]
[Thread 4269360352 (zombie) exited]
[Thread 4276200672 (zombie) exited]
[Thread 4282774752 (zombie) exited]
[Thread 4267238624 (zombie) exited]
[Thread 4249314528 (zombie) exited]
[Thread 4274386144 (zombie) exited]
[Thread 3252172000 (zombie) exited]
[New Thread 3252172000 (LWP 7434)]
[New Thread 4274386144 (LWP 7435)]
[New Thread 4249314528 (LWP 7436)]
[New Thread 4267238624 (LWP 7437)]
[New Thread 4282774752 (LWP 7438)]
[New Thread 4276200672 (LWP 7439)]
[New Thread 4269360352 (LWP 7440)]
[New Thread 4256212192 (LWP 7441)]
[New Thread 4287755488 (LWP 7442)]
[New Thread 4222350560 (LWP 7443)]
[New Thread 4252144864 (LWP 7444)]
[New Thread 4264531168 (LWP 7445)]
[New Thread 4266214624 (LWP 7446)]
[New Thread 4266804448 (LWP 7447)]
[New Thread 4236350688 (LWP 7456)]
[New Thread 4263888096 (LWP 7457)]
[New Thread 4264924384 (LWP 7458)]
[New Thread 4212823264 (LWP 7459)]
[New Thread 4286772448 (LWP 7460)]
[New Thread 4269163744 (LWP 7461)]
[New Thread 4225438944 (LWP 7462)]
[New Thread 4240418016 (LWP 7463)]
[New Thread 4260041952 (LWP 7464)]
[New Thread 4272682208 (LWP 7465)]
[New Thread 4293354720 (LWP 7466)]
[New Thread 4255622368 (LWP 7467)]
[Thread 4246598880 (zombie) exited]
[Thread 4232352992 (zombie) exited]
[Thread 4258469088 (zombie) exited]
[Thread 4242269408 (zombie) exited]
[New Thread 4242269408 (LWP 7473)]
[New Thread 4258469088 (LWP 7474)]
[New Thread 4232352992 (LWP 7475)]
[New Thread 4246598880 (LWP 7476)]
[New Thread 4267631840 (LWP 7477)]
[New Thread 4245812448 (LWP 7478)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 4254442720 (LWP 7200)]
0x104fbb98 in hp_delete_key (info=0x2ca50840, keyinfo=0x2ca25fd8, record=0x1948f370 "ï¿½PA>\001", 
    recpos=0x30d0c050 ".ï¿½Tï¿½\017ï¿½\230ï¿½RA\003\001", flag=0) at hp_delete.c:124
124     hp_delete.c: No such file or directory.
        in hp_delete.c
Current language:  auto; currently c

(gdb) bt
#0  0x104fbb98 in hp_delete_key (info=0x2ca50840, keyinfo=0x2ca25fd8, record=0x1948f370 "ï¿½PA>\001", 
    recpos=0x30d0c050 ".ï¿½Tï¿½\017ï¿½\230ï¿½RA\003\001", flag=0) at hp_delete.c:124
#1  0x104faa6c in heap_write (info=0x2ca50840, record=0x1948f370 "ï¿½PA>\001") at hp_write.c:80
#2  0x102ebfd0 in ha_heap::write_row (this=0x1948f290, buf=0x1948f370 "ï¿½PA>\001") at ha_heap.cc:180
#3  0x10257304 in end_update (join=0x195c01c0, join_tab=0x195c18c8, end_of_records=false) at sql_select.cc:10919
#4  0x10253da4 in evaluate_join_record (join=0x195c01c0, join_tab=0x195c1748, error=0, report_error=0xfb2b8c34 "")
    at sql_select.cc:9932
#5  0x102539d4 in sub_select (join=0x195c01c0, join_tab=0x195c1748, end_of_records=false) at sql_select.cc:9817
#6  0x102533bc in do_select (join=0x195c01c0, fields=0x0, table=0x1de64078, procedure=0x0) at sql_select.cc:9580
#7  0x10239af0 in JOIN::exec (this=0x195c01c0) at sql_select.cc:1358
#8  0x1023b720 in mysql_select (thd=0xfb2b81a8, rref_pointer_array=0xfb2b8550, tables=0x2e386f60, wild_num=0, 
    fields=@0xfb2b84b4, conds=0x2e3886f8, og_num=1, order=0x0, group=0x2e388878, having=0x0, proc_param=0x0, 
    select_options=2156153344, result=0x2e388900, unit=0xfb2b81f8, select_lex=0xfb2b8428) at sql_select.cc:1901
#9  0x10235818 in handle_select (thd=0xfb2b81a8, lex=0xfb2b81e8, result=0x2e388900, setup_tables_done_option=0)
    at sql_select.cc:238
#10 0x101f383c in mysql_execute_command (thd=0xfb2b81a8) at sql_parse.cc:2499
#11 0x101fcce0 in mysql_parse (thd=0xfb2b81a8, 
    inBuf=0x2e3869e8 "SELECT zielland AS SEA_country_code, MIN(preisadult) AS SEA_price_adult_min  FROM angebot.pauschal_A WHERE  paxmax >= 2  AND kontflug >= 2  AND paxmin <= 2  AND paxmin + paxzustadult >= 2  AND zimmera"..., 
    length=459) at sql_parse.cc:5695
#12 0x101f14c4 in dispatch_command (command=COM_QUERY, thd=0xfb2b81a8, 
    packet=0xfb271ea9 " SELECT zielland AS SEA_country_code, MIN(preisadult) AS SEA_price_adult_min  FROM angebot.pauschal_A WHERE  paxmax >= 2  AND kontflug >= 2  AND paxmin <= 2  AND paxmin + paxzustadult >= 2  AND zimmer"..., 
    packet_length=461) at sql_parse.cc:1736
#13 0x101f0c74 in do_command (thd=0xfb2b81a8) at sql_parse.cc:1522
#14 0x101efa88 in handle_one_connection (arg=0xfe92fc88) at sql_parse.cc:1165
#15 0x0ffbc434 in start_thread () from /lib/tls/libpthread.so.0
#16 0x0fdef6e0 in clone () from /lib/tls/libc.so.6

I send you another core and the output of the gdb64 you requested

Can you, please, send the SHOW CREATE TABLE and SHOW TABLE STATUS results for the angebot.pauschal_A table used? Is it possible to determine complete SQL statement that was performed:

#11 0x101fcce0 in mysql_parse (thd=0xfb2b81a8, 
    inBuf=0x2e3869e8 "SELECT zielland AS SEA_country_code, MIN(preisadult) AS
SEA_price_adult_min  FROM angebot.pauschal_A WHERE  paxmax >= 2  AND kontflug >=
2  AND paxmin <= 2  AND paxmin + paxzustadult >= 2  AND zimmera"...,

32-bit binary used, memory overallocation