Description:
Mysql server crashes randomly with a Signal 11 message. As it restart automatically after crach.
My MySQL configuration:
---------------------------------
[mysqld]
user = mysql
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
skip-external-locking
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=0
#bind-address=127.0.0.1
bind-address=0.0.0.0
max_connections=500
# Set timout values
wait_timeout=7200
#connect_timeout=10
interactive_timeout=240
# Set MyISAM parameters
key_buffer_size=512M
key_buffer=128M
myisam_sort_buffer_size = 64M
myisam_use_mmap=1
# Set InnoDB parameters
innodb_file_per_table
# When using innodb_file_per_table please consider the increased need for filehandles.
# Set innodb_open_files to provide at least one filehandle per table.
# You might need to up open_files_limit as well - this is also used to cache .frm files and MyISAM files, so it must be even larger.
#open_files_limit=32768
#innodb_open_files=2048
# Set buffer pool size to 50 - 80% of availiable memory.
# For mixed shared environment set max to 50% of memory.
innodb_buffer_pool_size=2048M
innodb_additional_mem_pool_size=20M
# Set innodb thread concurency limit (2*(num_cpus+num_disks))
# Set maximum 8, because MySQL versions up to 5.1 (included) have an issue for more than 8
innodb_thread_concurrency=16
# Set this value for less context switching
# Set this value for less context switching
innodb_concurrency_tickets=700
# Make max allowed packet as big as Postfix message size limit
# and Maia Mail size limit settings.
max_allowed_packet=15M
# Make innodb log file size as big as max allowed packet.
# http://dev.mysql.com/doc/refman/5.0/en/adding-and-removing.html
# Set the log file size to about 25% of the buffer pool size.
# Typically anything between 128M and 512M is a good value
innodb_log_file_size=15M
#innodb_log_file_size=256M
innodb_log_buffer_size=15M
innodb_lock_wait_timeout=100
# MySQL table and query cache settings
table_cache=1024
query_cache_type=1
query_cache_limit=8M
query_cache_size=256M
# Threads settings
thread_cache_size=256
# For more information for read_rnd_buffer_size: http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_read_rnd_buffer...
# For more information for read_buffer_size: http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_read_buffer_siz...
read_buffer_size=2M
read_rnd_buffer_size=4M
# Network settings
net_read_timeout=240
max_connect_errors=240
net_write_timeout=240
# Set buffer size for sort operations
sort_buffer_size=16M
# Set join buffer size for non optimized queries
join_buffer_size=16M
# Set MySQL Internal Temporary Tables settings
# For more information here: http://dev.mysql.com/doc/refman/5.0/en/internal-temporary-tables.html
tmp_table_size=64M
max_heap_table_size=64M
# Logging.
#log=/var/log/mysql/sql.log
#log_warnings=2
#long_query_time=10
#log_slow_queries=/var/log/mysql/slow-queries.log
#log_queries_not_using_indexes=1
# Set default character set and collation
collation_server=utf8_unicode_ci
character_set_server=utf8
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links=0
[mysql]
default-character-set=utf8
[mysqldump]
default-character-set=utf8
[mysqld_safe]
log-error=/var/log/mysql/mysqld.log
socket=/var/run/mysqld/mysqld.sock
nice=0
I am pasting latest entry in the error log file, below.
---------------------------------------------------------------
110527 12:36:19 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=501
max_threads=500
threads_connected=188
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9352224 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x7f5fc447d7c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f5fcafb9e58 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7f60a25320c9]
/usr/sbin/mysqld(handle_segfault+0x404) [0x7f60a223b554]
/lib/libpthread.so.0(+0xfb40) [0x7f60a1ab9b40]
/lib/libc.so.6(memcpy+0xe) [0x7f60a0594a9e]
/usr/sbin/mysqld(mi_mmap_pread+0x9c) [0x7f60a24f115c]
/usr/sbin/mysqld(_mi_read_static_record+0x61) [0x7f60a24e8d61]
/usr/sbin/mysqld(_mi_read_rnd_static_record+0x35e) [0x7f60a24e916e]
/usr/sbin/mysqld(ha_myisam::rnd_next(unsigned char*)+0x2d) [0x7f60a24d8b0d]
/usr/sbin/mysqld(rr_sequential(READ_RECORD*)+0x19) [0x7f60a2333bb9]
/usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x65) [0x7f60a22a79c5]
/usr/sbin/mysqld(+0x3bea7d) [0x7f60a22a8a7d]
/usr/sbin/mysqld(JOIN::exec()+0xbac) [0x7f60a22bd9ec]
/usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x142) [0x7f60a22b9682]
/usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x174) [0x7f60a22bf004]
/usr/sbin/mysqld(+0x35f8da) [0x7f60a22498da]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x516) [0x7f60a224d916]
/usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x3fb) [0x7f60a2252e2b]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xb2c) [0x7f60a225396c]
/usr/sbin/mysqld(do_command(THD*)+0xea) [0x7f60a225482a]
/usr/sbin/mysqld(handle_one_connection+0x23d) [0x7f60a22464ed]
/lib/libpthread.so.0(+0x7971) [0x7f60a1ab1971]
/lib/libc.so.6(clone+0x6d) [0x7f60a05f492d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x7f5fcc22b880 is an invalid pointer
thd->thread_id=202837
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
pure virtual method called
terminate called without an active exception
Fatal signal 6 while backtracing
110527 12:36:22 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Log scan progressed past the checkpoint lsn 35 444882366
110527 12:36:23 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 35 446909026
110527 12:36:23 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
110527 12:36:28 InnoDB: Started; log sequence number 35 446909026
110527 12:36:28 [Note] Event Scheduler: Loaded 0 events
110527 12:36:28 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.49-1ubuntu8.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
How to repeat:
There is no single query which causes DB to crash. DB has been running fine with
thousands of queries for days, and suddenly crashes with signal 11, with no change in query or load.
Suggested fix:
unknown
