Description:
Crash during high load on MySQL server.
In the first crash MySQL died with next output:
------------------------------------------------------------------------------------
InnoDB: Assertion failure in thread 32777
in file buf0lru.c line 827
InnoDB: Failing assertion: block->n_pointers == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html
InnoDB: about forcing recovery.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=59
max_connections=200
threads_connected=30
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 443390 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfe9eb98, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x809f182
0x82dceb8
0x8217b9c
0x82189a7
0x8215709
0x8216061
0x8210c95
0x8218a6a
0x82197f2
0x81ac289
0x81ac2ed
0x818e5dd
0x82da66c
0x8303faa
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow
instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
------------------------------------------------------------------------------------
Resolved stack trace is:
0x809f182 handle_segfault + 426
0x82dceb8 pthread_sighandler + 184
0x8217b9c buf_LRU_block_free_non_file_page + 184
0x82189a7 buf_LRU_block_free_hashed_page + 131
0x8215709 buf_LRU_search_and_free_block + 677
0x8216061 buf_LRU_get_free_block + 1417
0x8210c95 buf_page_init_for_read + 145
0x8218a6a buf_read_page_low + 186
0x82197f2 buf_read_ibuf_merge_pages + 170
0x81ac289 ibuf_contract_ext + 1245
0x81ac2ed ibuf_contract_for_n_pages + 41
0x818e5dd srv_master_thread + 793
0x82da66c pthread_start_thread + 220
0x8303faa thread_start + 4
After first crash we removed DB with drop command and created needed tablespaces again. MySQL died during load with next output:
------------------------------------------------------------------------------------
InnoDB: Assertion failure in thread 163881
in file buf0lru.c line 827
InnoDB: Failing assertion: block->n_pointers == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/mysql/en/Forcing_recovery.html
InnoDB: about forcing recovery.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=48
max_connections=200
threads_connected=28
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 443390 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=0xaf42d6d8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfa9dcb8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x809f182
0x82dceb8
0x8217b9c
0x82189a7
0x8215709
0x8216061
0x8210c95
0x8218a6a
0x8218f66
0x8218fda
0x820f056
0x81e306c
0x81c461c
0x81c4ea9
0x813a412
0x80ee5d9
0x80eda50
0x80edc09
0x80eda5c
0x80ed750
0x80de483
0x80df7bb
0x80dbfe1
0x80b0a50
0x80b7378
0x80aefd3
0x80ae8f3
0x80ade44
0x82da66c
0x8303faa
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow
instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0xb07044a8 is invalid pointer
thd->thread_id=725
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
InnoDB: Thread 122911 stopped in file btr0sea.c line 336
InnoDB: Thread 270359 stopped in file ../include/sync0sync.ic line 111
InnoDB: Thread 110620 stopped in file os0sync.c line 501
InnoDB: Thread 32777 stopped in file ../include/sync0sync.ic line 111
------------------------------------------------------------------------------------
Resolved stack trace is:
0x809f182 handle_segfault + 426
0x82dceb8 pthread_sighandler + 184
0x8217b9c buf_LRU_block_free_non_file_page + 184
0x82189a7 buf_LRU_block_free_hashed_page + 131
0x8215709 buf_LRU_search_and_free_block + 677
0x8216061 buf_LRU_get_free_block + 1417
0x8210c95 buf_page_init_for_read + 145
0x8218a6a buf_read_page_low + 186
0x8218f66 buf_read_ahead_random + 966
0x8218fda buf_read_page + 38
0x820f056 buf_page_get_gen + 634
0x81e306c btr_cur_search_to_nth_level + 1520
0x81c461c row_sel_try_search_shortcut_for_mysql + 64
0x81c4ea9 row_search_for_mysql + 1989
0x813a412 index_read__11ha_innobasePcPCcUi16ha_rkey_function + 446
0x80ee5d9 join_read_key__FP13st_join_table + 153
0x80eda50 sub_select__FP4JOINP13st_join_tableb + 200
0x80edc09 evaluate_join_record__FP4JOINP13st_join_tableiPc + 377
0x80eda5c sub_select__FP4JOINP13st_join_tableb + 212
0x80ed750 do_select__FP4JOINPt4List1Z4ItemP8st_tableP9Procedure + 508
0x80de483 exec__4JOIN + 1351
0x80df7bb mysql_select__FP3THDPPP4ItemP13st_table_listUiRt4List1Z4ItemP4ItemUiP8st_orderT7T5T7UlP13select_resultP18st_select_lex_unitP13s + 883
0x80dbfe1 handle_select__FP3THDP6st_lexP13select_resultUl + 193
0x80b0a50 mysql_execute_command__FP3THD + 1324
0x80b7378 mysql_parse__FP3THDPcUi + 288
0x80aefd3 dispatch_command__F19enum_server_commandP3THDPcUi + 1747
0x80ae8f3 do_command__FP3THD + 195
0x80ade44 handle_one_connection + 764
0x82da66c pthread_start_thread + 220
0x8303faa thread_start + 4
------------------------------------------------------------------------------------
MySQL server 5.0.18-standard-log installed from static binaries.
The computer tested with different HW tests and pass all ok. So this isn't HW problem.
How to repeat:
Currently i research how it can be repeated, but no sexact scenario available
