Bug #65473 Random crashes on MYSQL server
Submitted: 31 May 2012 14:21 Modified: 30 Jun 2012 15:01
Reporter: Arnold Human Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Errors Severity:S1 (Critical)
Version:Ver 14.14 Distrib 5.5.9 OS:Linux (Red Hat Enterprise Linux Server release 5.3 (Tikanga) x86_64)
Assigned to: CPU Architecture:Any

[31 May 2012 14:21] Arnold Human
Description:
Hi. We've noticed that our MYSQL database keeps crashing randomly on a Signal 11 error. I've found some references to this problem, but nothing related to the version we are running, and most of the posts, have been old.

The exact error I am getting is:

120531 14:41:36 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=39
max_threads=1024
thread_count=33
connection_count=33
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2248056 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x1c02fde0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x4a6df0d8 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x94e6a9]
/usr/sbin/mysqld(handle_segfault+0x352)[0x4fc5e2]
/lib64/libpthread.so.0[0x36ce20e4c0]
/usr/sbin/mysqld(_ZN5Field10make_fieldEP10Send_field+0x19)[0x66b1a9]
/usr/sbin/mysqld(_ZN10Item_field10make_fieldEP10Send_field+0x26)[0x68d986]
/usr/sbin/mysqld(_ZN8Protocol24send_result_set_metadataEP4ListI4ItemEj+0x35c)[0x50bd7c]
/usr/sbin/mysqld(_ZN11select_send24send_result_set_metadataER4ListI4ItemEj+0x1d)[0x54886d]
/usr/sbin/mysqld(_ZN28Select_fetch_protocol_binary24send_result_set_metadataER4ListI4ItemEj+0x2e)[0x58547e]
/usr/sbin/mysqld(_ZN19Materialized_cursor4openEP4JOIN+0xf4)[0x7803a4]
/usr/sbin/mysqld(_Z17mysql_open_cursorP3THDP13select_resultPP18Server_side_cursor+0x18c)[0x78023c]
/usr/sbin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x370)[0x585e40]
/usr/sbin/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x64)[0x588f14]
/usr/sbin/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x1b0)[0x589530]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xc89)[0x57b899]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xc4)[0x57c0f4]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x124)[0x613aa4]
/usr/sbin/mysqld(handle_one_connection+0x54)[0x613b94]
/lib64/libpthread.so.0[0x36ce206367]
/lib64/libc.so.6(clone+0x6d)[0x36cd6d30ad]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x1bdc8f20 = SHOW SESSION VARIABLES
thd->thread_id=47
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120531 14:41:36 mysqld_safe Number of processes running now: 0
120531 14:41:36 mysqld_safe mysqld restarted
120531 14:41:36 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
120531 14:41:36  InnoDB: Using Linux native AIO
120531 14:41:36  InnoDB: Initializing buffer pool, size = 8.0G
120531 14:41:37  InnoDB: Completed initialization of buffer pool
120531 14:41:37  InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 4277754574731
120531 14:41:38  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 4277759817216
InnoDB: Doing recovery: scanned up to log sequence number 4279215050335
120531 14:41:55  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 4708105, file name ./o9902-cordys-rep.006462

The crashes randomly happens, today for example it happened 3 times in 15 minutes. As we are getting these errors in a production environment, the impact of this is massive! 

We have 32G RAM on the server and 8 CPUs.

How to repeat:
Not sure
[31 May 2012 14:21] Arnold Human
Our my.cnf looks like:

[mysqld]
datadir=/mysql1/mysql/data
max_allowed_packet=50M
log-bin=o9902-cordys-rep
server-id=1
sync_binlog=1
binlog_format=MIXED
max_connections=1024
max_user_connections=1024
innodb_file_per_table=1

## SET BY AFH 2012-05-24
thread_concurrency=8             # Higher values does not make a difference
innodb_buffer_pool_size=8G      # for dedicated server set it to 80% of your RAM
innodb_log_file_size=1G          # set it to 25% of innodb_buffer_pool_size
innodb_flush_log_at_trx_commit=0 # When you want true ACID set it to 1, but slows mysql down with factor 2.
innodb_thread_concurrency=64     # do not set it higher because of thread thrashing
innodb_commit_concurrency=0      # total concurrent
max_heap_table_size=128M
tmp_table_size=128M

# Set the SQL mode to strict
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"

default-storage-engine=INNODB

# Query cache is used to cache SELECT results and later return them
# without actual executing the same query once again. Having the query
# cache enabled may result in significant speed improvements, if your
# have a lot of identical queries and rarely changing tables. See the
# "Qcache_lowmem_prunes" status variable to check if the current value
# is high enough for your load.
# Note: In case your tables change very often or if your queries are
# textually different every time, the query cache may result in a
# slowdown instead of a performance improvement.
query_cache_size=29M

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
table_cache=500

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
#tmp_table_size=33M

# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=11

#*** INNODB Specific options ***

# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
#skip-innodb

# Additional memory pool that is used by InnoDB to store metadata
# information.  If InnoDB requires more memory for this purpose it will
# start to allocate it from the OS.  As this is fast enough on most
# recent operating systems, you normally do not need to change this
# value. SHOW INNODB STATUS will display the current amount used.
innodb_additional_mem_pool_size=2M

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
#innodb_flush_log_at_trx_commit=1

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
innodb_log_buffer_size=1M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
#innodb_buffer_pool_size=85M

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
#innodb_log_file_size=17M

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
#innodb_thread_concurrency=18
[31 May 2012 15:01] Valeriy Kravchuk
Try to decrease these values in my.cnf:

max_heap_table_size=16M # was 128M
tmp_table_size=16M # was 128M

and check if this will change anything.

If you really have up to 1000 concurrent connections and they use internal temporary tables a lot, then out of memory situation is quite possible.

If this will not help and you feel the reason is some bug in MySQL, please, upgrade to recent version, 5.5.25, first (as we do not fix bugs that are not repeatable in current versions).
[1 Jun 2012 8:58] MySQL Verification Team
Hi!

This looks like a duplicate of:

http://bugs.mysql.com/bug.php?id=56115
Bug 11763413: 56115: SELECT DOESN'T WORK IN PREPARED STATEMENTS WITH CURSOR PROTOCOL

btw, 5.5.25 is the current version of mysql.
[1 Jul 2012 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".