Bug #25698 mysqld got signal 11 and restart frequently
Submitted: 18 Jan 2007 17:08 Modified: 12 Feb 2007 8:58
Reporter: Chaoqun Yie Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.0.33 OS:Linux (RedHat AS4)
Assigned to: CPU Architecture:Any

[18 Jan 2007 17:08] Chaoqun Yie
Description:
My server is AMD64+4G . I've got MySQL 5.0.33 and RedHat AS4 and have some issues with it frequently restart.
Does anyone know what could cause this or how it could be solved?

(1)
Hardware: AMD64*2/4G Memory 
Software: RedHat 32bit AS4 "Linux 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686 athlon i386 GNU/Linux"

Mysql is compiled with options:
CFLAGS="-O3" CXX=gcc CXXFLAGS="-O3 -felide-constructors -fno-exceptions -fno-rtti" ./configure --prefix=/usr/local/mysql-5.0  --enable-assembler --with-mysqld-ldflags=-all-static

(2)
Here is the error log:

mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=402653184
read_buffer_size=2093056
max_used_connections=7
max_connections=100
threads_connected=6
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 802415 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x8ad3e48
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbe9fdb98, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x80cd9e3
0x833f60c
0x8362364
0x812278d
0x812b153
0x8131199
0x8131861
0x80e33be
0x80ec0bb
0x80ecb8b
0x80ee06a
0x8339f45
0x8369bba
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow
 instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x8ab7960 = select count(*) as length from iask_tbl_ques_answ whe
re iQuesID='268142'
thd->thread_id=265
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
070117 00:19:14  mysqld restarted

(3)
Here is the Stacktrace from the crash:

0x80cd9e3 handle_segfault + 603
0x833f60c __pthread_sighandler + 144
0x8362364 memcpy + 40
0x812278d _Z20make_join_statisticsP4JOINP13st_table_listP4ItemP16st_dynamic_array + 8317
0x812b153 _ZN4JOIN8optimizeEv + 1415
0x8131199 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sel + 141
0x8131861 _Z13handle_selectP3THDP6st_lexP13select_resultm + 241
0x80e33be _Z21mysql_execute_commandP3THD + 1206
0x80ec0bb _Z11mysql_parseP3THDPcj + 527
0x80ecb8b _Z16dispatch_command19enum_server_commandP3THDPcj + 2491
0x80ee06a handle_one_connection + 1606
0x8339f45 pthread_start_thread + 201
0x8369bba clone + 106

How to repeat:
When the server is under heavy load.
[22 Jan 2007 13:40] Valeriy Kravchuk
Thank you for a problem report. Please, send your my.cnf file content. Upload also SHOW GLOBAL STATUS results obtained when your server is under load.
[27 Jan 2007 3:52] Chaoqun Yie
show global status

Attachment: global_status.txt (text/plain), 11.47 KiB.

[27 Jan 2007 3:53] Chaoqun Yie
my.cnf

Attachment: my.cnf.cnf (application/octet-stream, text), 4.96 KiB.

[8 Feb 2007 23:04] johnny slakva
it seems like we have same problem. maybe our details will make some help... sometimes it dies with query information, sometimes even without it; actual sql query, when present, is not repeating.

if it can be of help, i can try resolving stack from some other similar crashes (i think we have about 10 in logs so far), or what other information can you need if our case is related to same bug.

we are using ndbcluster.
mysql 5.0.33.
OS: redhat enterprise linux 3.
2.4.21-47.0.1.EL #1 Fri Oct 13 18:04:55 EDT 2006 i686 i686 i386 GNU/Linux

error message:
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=2093056
max_used_connections=17
max_connections=500
threads_connected=6
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 2062380 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0xb505e158
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x17178c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8190b20
0x83ab78e
0x83bbe27
0x83a2867
0x839f714
0x824dd12
0x8259248
0x824e9e2
0x82508f3
0x81e7d21
0x81e7050
0x81e71fb
0x81e6ffd
0x81e6c07
0x81d6a0e
0x81d7dc5
0x81d3c16
0x81a6d79
0x81addfe
0x81a538b
0x81a4ebd
0x81a43af
0x908dd8
0x717d1a
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xb9e9e40 = select art.url, art.title, art.popular, art.titleLinesInRelated
        from similarities s
        left join articles art on s.artid_to = art.id
        where artid_from = 1900 and art.completed=1
        order by art.id desc
thd->thread_id=110773
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
070208 13:59:12  mysqld restarted

resolved stack: 
0x8190b20 handle_segfault + 416
0x83ab78e _ZNK13NdbDictionary5Table9getColumnEPKc + 30
0x83bbe27 _ZN7NdbBlob9atPrepareEP14NdbTransactionP12NdbOperationPK13NdbColumnImpl + 231
0x83a2867 _ZN12NdbOperation13getBlobHandleEP14NdbTransactionPK13NdbColumnImpl + 119
0x839f714 _ZN12NdbOperation13getBlobHandleEj + 68
0x824dd12 _ZN13ha_ndbcluster13get_ndb_valueEP12NdbOperationP5FieldjPc + 210
0x8259248 _ZN13ha_ndbcluster17define_read_attrsEPcP12NdbOperation + 184
0x824e9e2 _ZN13ha_ndbcluster7pk_readEPKcjPc + 178
0x82508f3 _ZN13ha_ndbcluster10index_readEPcPKcj16ha_rkey_function + 275
0x81e7d21 _Z13join_read_keyP13st_join_table + 145
0x81e7050 _Z10sub_selectP4JOINP13st_join_tableb + 272
0x81e71fb _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 363
0x81e6ffd _Z10sub_selectP4JOINP13st_join_tableb + 189
0x81e6c07 _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 295
0x81d6a0e _ZN4JOIN4execEv + 1502
0x81d7dc5 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sel + 309
0x81d3c16 _Z13handle_selectP3THDP6st_lexP13select_resultm + 374
0x81a6d79 _Z21mysql_execute_commandP3THD + 681
0x81addfe _Z11mysql_parseP3THDPcj + 318
0x81a538b _Z16dispatch_command19enum_server_commandP3THDPcj + 1147
0x81a4ebd _Z10do_commandP3THD + 141
0x81a43af handle_one_connection + 655
0x908dd8 (?)
0x717d1a (?)
[9 Feb 2007 11:01] Chaoqun Yie
It seems that I have gotten the solution of this problem. I changed the OS to the x86-64 version of AS4. Now, it seems mysql works ok.
[12 Feb 2007 8:58] Valeriy Kravchuk
If changing OS to 64-bit one helped, then I could be just out-of-memory issue.