Bug #19518 Server crash on start with signal 11
Submitted: 3 May 2006 14:12 Modified: 1 Jul 2006 13:02
Reporter: Dimitrij HIlt Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:mysql-standard-5.0.21-linux-i686-glibc23 OS:Linux (Linux Debian/libc-2.3.2/2.6.12)
Assigned to: CPU Architecture:Any

[3 May 2006 14:12] Dimitrij HIlt
Description:
After update from mysql-standard-5.0.20-linux-i686-glibc23 to mysql-standard-5.0.21-linux-i686-glibc23 ( both official binarys from mysql download server) and new start server dies with sig 11.

Log:
060503 15:56:37  InnoDB: Starting shutdown...
060503 15:56:41  InnoDB: Shutdown completed; log sequence number 3 3902967651
060503 15:56:41 [Note] /usr/local/mysql5/bin/mysqld: Shutdown complete

060503 15:56:42  InnoDB: Started; log sequence number 3 3902967651
060503 15:56:42 [Note] /usr/local/mysql5/bin/mysqld: ready for connections.
Version: '5.0.21-standard-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 330
6  MySQL Community Edition - Standard (GPL)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=14
max_connections=240
threads_connected=7
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 219134
 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0xa3c7ab80
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xa3bfc768, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8171f24
0xffffe420
(nil)
0x81fe3e3
0x81c3983
0x81badba
0x81b1dbe
0x81b5056
0x81b10ef
0x8186367
0x818d63e
0x8184c82
0x81847ad
0x8183cb5
0xb7f93b63
0xb7ecc18a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow 
instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xa3814f30  is invalid pointer
thd->thread_id=264
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Stack resolver says:
0x8171f24 handle_segfault + 356
0xffffe420 _end + -140125424
(nil)
0x81fe3e3 _ZN10SQL_SELECT17test_quick_selectEP3THD6BitmapILj64EEyyb + 1283
0x81c3983 _Z22get_quick_record_countP3THDP10SQL_SELECTP8st_tablePK6BitmapILj64EEy + 59
0x81badba _Z20make_join_statisticsP4JOINP13st_table_listP4ItemP16st_dynamic_array + 2550
0x81b1dbe _ZN4JOIN8optimizeEv + 850
0x81b5056 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_mP13select_resultP18st_select_lex_unitP13st_sel + 130
0x81b10ef _Z13handle_selectP3THDP6st_lexP13select_resultm + 267
0x8186367 _Z21mysql_execute_commandP3THD + 571
0x818d63e _Z11mysql_parseP3THDPcj + 306
0x8184c82 _Z16dispatch_command19enum_server_commandP3THDPcj + 1178
0x81847ad _Z10do_commandP3THD + 129
0x8183cb5 handle_one_connection + 621
0xb7f93b63 _end + -1348521389
0xb7ecc18a _end + -1349339014
rdb206:/usr/local/mysql-standard-5.0.21-linux-i686-glibc23# 
rdb206:/usr/local/mysql-standard-5.0.21-linux-i686-glibc23# ./bin/resolve_stack_dump -s /tmp/mysqld.sym -n /tmp/mysqld.stack
0xa3c7ab80 _end + -1687313808
0x8171f24 handle_segfault + 356
0xffffe420 _end + -140125424
(nil)
0x81fe3e3 _ZN10SQL_SELECT17test_quick_selectEP3THD6BitmapILj64EEyyb + 1283
0x81c3983 _Z22get_quick_record_countP3THDP10SQL_SELECTP8st_tablePK6BitmapILj64EEy + 59
0x81badba _Z20make_join_statisticsP4JOINP13st_table_listP4ItemP16st_dynamic_array + 2550
0x81b1dbe _ZN4JOIN8optimizeEv + 850
0x81b5056 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_mP13select_resultP18st_select_lex_unitP13st_sel + 130
0x81b10ef _Z13handle_selectP3THDP6st_lexP13select_resultm + 267
0x8186367 _Z21mysql_execute_commandP3THD + 571
0x818d63e _Z11mysql_parseP3THDPcj + 306
0x8184c82 _Z16dispatch_command19enum_server_commandP3THDPcj + 1178
0x81847ad _Z10do_commandP3THD + 129
0x8183cb5 handle_one_connection + 621
0xb7f93b63 _end + -1348521389
0xb7ecc18a _end + -1349339014

How to repeat:
Just update and try? I dont know anythink aout query and have a lot of customer databases on this system, so i dont know anythink about DB scheme.
[5 May 2006 6:08] Dimitrij HIlt
Doesn't happens with mysql-standard-5.0.21-linux-i686 statically linked.
[6 May 2006 8:09] Dimitrij HIlt
Staticaly compiled mysql-standard-5.0.21-linux-i686 has just another Problem and crashes too:

mysql.err:
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=12
max_connections=240
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 219134
 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0xabe69738
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbf79c8e8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x809f102
0x82ef958
0x812006e
0x8122dff
0x81236b4
0x8123095
0x8120a43
0x80dfa99
0x80e0b3a
0x80dce77
0x80df959
0x80dc141
0x80b0ae5
0x80b72ad
0x80af086
0x80ae923
0x80ade64
0x82ed10c
0x832e4ea
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow 
instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8a9d800 = SELECT DISTINCT * FROM janberlin_posts  WHERE 1=1 AND 
(post_date_gmt <= '2006-05-06 05:08:59' AND id NOT IN (-1,4,5) ) AND (post_statu
s = "publish") AND post_status != "attachment" GROUP BY  janberlin_posts.ID  ORD
ER BY post_date DESC LIMIT 0, 10
thd->thread_id=12410
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Resolve:
0x809f102 handle_segfault + 430
0x82ef958 pthread_sighandler + 184
0x812006e last__7SEL_ARG + 6
0x8122dff get_func_mm_tree__FP13st_qsel_paramP9Item_funcP5FieldP4Item11Item_resultb + 623
0x81236b4 get_mm_tree__FP13st_qsel_paramP4Item + 1700
0x8120a43 test_quick_select__10SQL_SELECTP3THDGt6Bitmap1Ui64UxUxb + 1419
0x80dfa99 get_quick_record_count__FP3THDP10SQL_SELECTP8st_tablePCt6Bitmap1Ui64Ux + 69
0x80e0b3a make_join_statistics__FP4JOINP13st_table_listP4ItemP16st_dynamic_array + 4194
0x80dce77 optimize__4JOIN + 971
0x80df959 mysql_select__FP3THDPPP4ItemP13st_table_listUiRt4List1Z4ItemP4ItemUiP8st_orderT7T5T7UlP13select_resultP18st_select_lex_unitP13s + 801
0x80dc141 handle_select__FP3THDP6st_lexP13select_resultUl + 225
0x80b0ae5 mysql_execute_command__FP3THD + 1329
0x80b72ad mysql_parse__FP3THDPcUi + 281
0x80af086 dispatch_command__F19enum_server_commandP3THDPcUi + 1878
0x80ae923 do_command__FP3THD + 195
0x80ade64 handle_one_connection + 764
0x82ed10c pthread_start_thread + 220
0x832e4ea thread_start + 4
[6 May 2006 8:18] Dimitrij HIlt
On all crashes ( different databases ) is the stack trace allways similar to reported.
[12 May 2006 11:41] Valeriy Kravchuk
Let's concentrate on the query from the last crash (although, upgrade to libc-2.3.5-3 at least is worth trying, because of http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=314408):

SELECT DISTINCT * FROM janberlin_posts  
WHERE 
1=1 AND
(post_date_gmt <= '2006-05-06 05:08:59' AND id NOT IN (-1,4,5) )  AND
(post_status = "publish") AND 
post_status != "attachment" GROUP BY  janberlin_posts.ID 
ORDER BY post_date DESC LIMIT 0, 10

So, please, send the SHOW CREATE TABLE and SHOW TABLE STATUS results for that janberlin_posts table.
[12 May 2006 11:54] Dimitrij HIlt
janberlin_posts definitions.

Attachment: janberlin_posts.txt (text/plain), 9.50 KiB.

[12 May 2006 12:00] Dimitrij HIlt
Hi,

table status and creation command are now uploaded as attachment.

I cann't upgrade libc6 because we will get everytime clean Deabian distribution and debian has 2.3.2.ds1-22sarge3 as libc6 version. And what is with staticaly linked binarys from mysql.com? Why we got same problem with this binarys?

Dimi
[12 May 2006 12:22] Dimitrij HIlt
libc6 bug was fixed in libc6-2.3.2.ds1-22sarge2, but it is only amd64 relavant. We have Intel Xeon Hardware.

Dimi
[8 Jun 2006 23:41] Dimitrij HIlt
alter table alter table janberlin_posts type=myisam; solves this problem, but mysqlcheck says this table is ok.

Dimi
[9 Jun 2006 6:04] Anders Henke
Ouch. Looks pretty much like some error in the MyISAM storage engine, if "alter table" to the same type fixes this issue.
Note: the database has been created using a "general available" 5.0-release (the server has always been running 5.0), so I doubt that this is some 4.x->5.0 migration issue.
[9 Jun 2006 6:20] Anders Henke
I think that this is an issue in the MyISAM engine, so I've uploaded the untouched MyISAM-tables for janberlin_posts.* as bug19518.tar.gz to the "secret" area of ftp://ftp.mysql.com/pub/mysql/upload/.

Running the given query in 5.0.19 (mysql-standard-5.0.19-linux-i686) works fine, running in 5.0.22 (mysql-standard-5.0.22-linux-i686) immediately crashes the database server. After recreating the table (either by "ALTER TABLE" to the same type or by "CREATE TABLE new select * from janberlin_posts") results in a new table which won't crash the server anymore.

Anders
[1 Jul 2006 13:02] Valeriy Kravchuk
Anders,

I was not able to repeat the crash with your tables uploaded and copied, on 5.0.24-BK. I'll post results of select as private comment, for Dimitrij to check.