Description:
Our Mysql server crashes once a day with the error message below.
Server info:
- 2x Intel XEON
- 12GB mem
Kernel:
2.6.9-42.0.3.ELsmp
Installed packages:
MySQL-client-standard-4.1.21-0.rhel4
MySQL-server-standard-4.1.21-0.rhel4
perl-DBD-MySQL-2.9004-3.1
MySQL-shared-standard-4.1.21-0.rhel4
-----
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=100663296
read_buffer_size=2093056
max_used_connections=153
max_connections=500
threads_connected=24
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 4192300 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0x4631d178
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x45a593c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x812b1e6
0x66d898
0x82ef9f5
0x8240abc
0x81b2e9d
0x81aa589
0x81a33c6
0x81a599d
0x8168ca9
0x816907b
0x8175b85
0x810ce38
0x8109c98
0x810aa41
0x80e9541
0x80ed0cd
0x80ed04d
0x8168d9b
0x8168d00
0x816907b
0x8175173
0x8176a32
0x81770a6
0x8140f42
0x8142b88
0x81435a6
0x8143e47
0x814455d
0x667371
0x552ffe
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x9b2da78 = select distinct
p.ID
from
PXX_PZZ p
left join
CILL_AMS_PZZ cp
on
p.ID = cp.PXX_PZZ_ID
where
(cp.CILL_AMS_ID IN ( select
CILL_AMS_ID
from
MAS_MOP_AMS
where
MAS_MOP_ID IN (10005,10005))
or
cp.CILL_AMS_ID IN ( select
CILL_AMS_ID
from
MAS_MOP
where
thd->thread_id=4079969
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
Number of processes running now: 0
061211 16:09:12 mysqld restarted
061211 16:09:13 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
061211 16:09:15 InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 8 4233933131.
InnoDB: Doing recovery: scanned up to log sequence number 8 4234139290
061211 16:09:15 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 4
6 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98
99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 820, file name server02-bin.000042
InnoDB: Last MySQL binlog file position 0 271620648, file name ./server01-bin.000037
061211 16:09:17 InnoDB: Flushing modified pages from the buffer pool...
---- resolve_stack_dump output:
0x812b1e6 handle_segfault + 610
0x66d898 (?)
0x82ef9f5 mtr_memo_slot_release + 2653
0x8240abc row_search_for_mysql + 7081
0x81b2e9d _ZN11ha_innobase13general_fetchEPcjj + 81
0x81aa589 _ZN7handler15read_range_nextEv + 53
0x81a33c6 _ZN12QUICK_SELECT8get_nextEv + 234
0x81a599d _Z8rr_quickP14st_read_record + 23
0x8168ca9 _Z10sub_selectP4JOINP13st_join_tableb + 99
0x816907b _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 493
0x8175b85 _ZN4JOIN4execEv + 4741
0x810ce38 _ZN30subselect_single_select_engine4execEv + 104
0x8109c98 _ZN14Item_subselect4execEv + 34
0x810aa41 _ZN17Item_in_subselect7val_intEv + 21
0x80e9541 _ZN17Item_in_optimizer7val_intEv + 73
0x80ed0cd _ZN12Item_cond_or7val_intEv + 59
0x80ed04d _ZN13Item_cond_and7val_intEv + 59
0x8168d9b _Z10sub_selectP4JOINP13st_join_tableb + 341
0x8168d00 _Z10sub_selectP4JOINP13st_join_tableb + 186
0x816907b _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 493
0x8175173 _ZN4JOIN4execEv + 2163
0x8176a32 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_mP13select_resultP18st_select_lex_unitP13st_sel + 184
0x81770a6 _Z13handle_selectP3THDP6st_lexP13select_result + 292
0x8140f42 _Z21mysql_execute_commandP3THD + 12946
0x8142b88 _Z11mysql_parseP3THDPcj + 166
0x81435a6 _Z16dispatch_command19enum_server_commandP3THDPcj + 2472
0x8143e47 _Z10do_commandP3THD + 127
0x814455d handle_one_connection + 1581
0x667371 (?)
0x552ffe (?)
How to repeat:
Still trying to reproduce.
At this moment it will happen once a day.
