Description:
The server has been running over 330 days without any problems. Then all the sudden mysql crashes on RHEL4 (all patches applied) while trying to allocated memory. (There was over 4GB swap memory available, total of 4GB of system memory). Thanks for building in the auto restart feature!
080522 16:29:13 InnoDB: Error: cannot allocate 1064960 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 1801141382 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: We keep retrying the allocation for 60 seconds...
080522 16:33:37 InnoDB: Error: cannot allocate 1048576 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 1801080046 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: We keep retrying the allocation for 60 seconds...
080522 16:59:46 InnoDB: Error: cannot allocate 1048576 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 1801026238 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: We keep retrying the allocation for 60 seconds...
080522 18:05:54 InnoDB: Error: cannot allocate 1064960 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 1801088974 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: We keep retrying the allocation for 60 seconds...
080522 18:06:54 InnoDB: We now intentionally generate a seg fault so that
InnoDB: on Linux we get a stack trace.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=277
max_connections=400
threads_connected=78
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 878588 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x6ac64e7c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x8136e04
0x9aba98
0x94f1e28
0x83ab410
0x836f373
0x828799f
0x9a53cc
0x82f1ae
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
How to repeat:
Really don't know. Ulimits are pretty much linux defaults. We running innodb tables only.
[root@web2 log]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 73728
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
InnoDB settings are as follows:
max_heap_table_size=100M;
max_connections = 400
innodb_buffer_pool_size = 1024M
innodb_additional_mem_pool_size = 128M
innodb_log_file_size = 5242880
innodb_log_buffer_size = 8M
Here is the stack trace:
[root@web2 mysql]# resolve_stack_dump -s /tmp/mysqld.sym -n mysqld.stack
0x8136e04 handle_segfault + 610
0x9aba98 (?)
0x94f1e28 _end + 15886068
0x83ab410 os_aio_simulated_handle + 1872
0x836f373 fil_aio_wait + 542
0x828799f srv_parse_log_group_home_dirs + 207
0x9a53cc (?)
0x82f1ae (?)
