Description:
I tried to run SysBench with 1000 threads
and MySQL crashed during test.
(quadxeon, 4x 2GHz Xeon, MemTotal: 4024536 kB
2.6.9-1.648_EL.rootsmp kernel)
SysBench output:
Threads started!
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err5 Out of memory (Needed 720 bytes)
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err1105 Unknown error
database error, exiting...
failed to execute mysql_stmt_execute(): Err2013 Lost connection to MySQL server during query
MySQL log-error:
/mnt/test2/vt/mysql/bin/mysqld: ready for connections.
Version: '4.1.10-standard' socket: '/tmp/mysql.sock' port: 3306 MySQL Community Edition - Standard (GPL)
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=1048576
read_buffer_size=1044480
max_used_connections=1000
max_connections=16000
threads_connected=1000
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 592259 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
You seem to be running 32-bit Linux and have 1000 concurrent connections.
If you have not changed STACK_SIZE in LinuxThreads and built the binary
yourself, LinuxThreads is quite likely to steal a part of the global heap for
the thread stack. Please read http://www.mysql.com/doc/en/Linux.html
thd=0xc20fa80
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb8e7d9b8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x808b193
0x82deab8
0x81833b0
0x8101880
0x80f1243
0x80f0c9f
0x80eddca
0x80bdfd6
0x80bedea
0x80bb829
0x80bded6
0x80baf26
0x809ae3a
0x80d468b
0x80d445d
0x80999a1
0x8099438
0x8098b07
0x82dc26c
0x8305bfa
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x115c4898 = SELECT sum(c) from sbtest where id between ? and ?
thd->thread_id=898
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
050311 7:31:04 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
==== my.cnf =====
max_connections = 16000
max_connect_errors = 10
table_cache = 2048
#external-locking
max_allowed_packet = 1M
binlog_cache_size = 1M
max_heap_table_size = 64M
sort_buffer_size = 8M
join_buffer_size = 1M
thread_cache = 8
# You should try [number of CPUs]*(2..4) for thread_concurrency
thread_concurrency = 16
thread_stack = 256K
query_cache_size = 0M
query_cache_limit = 2M
ft_min_word_len = 4
#memlock
default_table_type = MYISAM
transaction_isolation = REPEATABLE-READ
tmp_table_size = 64M
#*** MyISAM Specific options
key_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 1M
bulk_insert_buffer_size = 1M
myisam_sort_buffer_size = 1M
myisam_max_sort_file_size = 1M
myisam_max_extra_sort_file_size = 1M
myisam_repair_threads = 1
skip-locking
query_cache_size = 0
server-id = 1
innodb_status_file=0
innodb_data_home_dir = /mnt/test2/vt/mysql/data
innodb_data_file_path = ibdata1:100M:autoextend
#innodb_data_file_path = ibdata1:2G;ibdata2:2G;ibdata3:2G:autoextend
innodb_log_group_home_dir = /mnt/test2/vt/mysql/data
#innodb_log_group_home_dir = /usr/local/mysql/data
#innodb_log_arch_dir = /usr/local/mysql/data/
innodb_buffer_pool_size = 1800M
# 1500M - for 5 - 10 W
# 2520M
innodb_additional_mem_pool_size = 20M
# Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 1G
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 300
innodb_locks_unsafe_for_binlog = 1
innodb_thread_concurrency = 1000
stack trace:
0x808b193 handle_segfault + 423
0x82deab8 pthread_sighandler + 184
0x81833b0 row_sel_convert_mysql_key_to_innobase + 1120
0x8101880 records_in_range__11ha_innobaseUiP12st_key_rangeT2 + 336
0x80f1243 check_quick_keys__FP13st_qsel_paramUiP7SEL_ARGPcUiT3Ui + 1339
0x80f0c9f check_quick_select__FP13st_qsel_paramUiP7SEL_ARG + 103
0x80eddca test_quick_select__10SQL_SELECTP3THDGt6Bitmap1Ui64UxUlb + 1230
0x80bdfd6 get_quick_record_count__FP3THDP10SQL_SELECTP8st_tablePCt6Bitmap1Ui64Ul + 62
0x80bedea make_join_statistics__FP4JOINP13st_table_listP4ItemP16st_dynamic_array + 3558
0x80bb829 optimize__4JOIN + 457
0x80bded6 mysql_select__FP3THDPPP4ItemP13st_table_listUiRt4List1Z4ItemP4ItemUiP8st_orderT7T5T7UlP13select_resultP18st_select_lex_unitP13s + 742
0x80baf26 handle_select__FP3THDP6st_lexP13select_result + 150
0x809ae3a mysql_execute_command__FP3THD + 1382
0x80d468b execute_stmt__FP3THDP18Prepared_statementP6Stringb + 155
0x80d445d mysql_stmt_execute__FP3THDPcUi + 333
0x80999a1 dispatch_command__F19enum_server_commandP3THDPcUi + 1373
0x8099438 do_command__FP3THD + 188
0x8098b07 handle_one_connection + 615
0x82dc26c pthread_start_thread + 220
0x8305bfa thread_start + 4
How to repeat:
1. run MySQL with my.cnf
2. prepare sysbench
./sysbench --test=oltp --oltp-table-size=1000000 prepare
3. run sysbench with 1000 threads:
./sysbench --test=oltp --num-threads=1000 --oltp-table-size=1000000 --max-requests=0 --max-time=180 run
