MySQL Bugs: #57652: InnoDB memory crash when restarting... sometimes...

Bug #57652	InnoDB memory crash when restarting... sometimes...
Submitted:	22 Oct 2010 7:58	Modified:	1 Dec 2010 22:17
Reporter:	Fabien LESTIENNE	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	5.1.37	OS:	Linux
Assigned to:		CPU Architecture:	Any

Description:
Hi there everybody,

We have some random memory crash with our MySQL Server. It runs with InnoDB engine on an Ubuntu 64 bits server with 16 Gb RAM.

Our MySQL server is restarting everynight (stopping at 1:45 AM and starting at 2:15 AM). Recently, it started crashing randomly when restarting (something like once in a week). It seems to appear when free memory is low (less than 2 Gb) even if inactive memory is high (more than 10 Gb).

Here is the error log :

101012  2:15:01  InnoDB: Error: cannot allocate 8589950976 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 49601872 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: Note that in most 32-bit computers the process
InnoDB: memory space is limited to 2 GB or 4 GB.
InnoDB: We keep retrying the allocation for 60 seconds...
InnoDB: Fatal error: cannot allocate the memory for the buffer pool

Here are some info on our system :

OS version : Linux 2.6.24-24-server x86_64 GNU/Linux
MySQL version : mysql Ver 14.14 Distrib 5.1.37, for debian-linux-gnu (x86_64) using  EditLine wrapper

vmstat -saSM :
        16071 M total memory
        11169 M used memory
         2665 M active memory
         8161 M inactive memory
         4902 M free memory
         1032 M buffer memory
         3724 M swap cache
         2999 M total swap
            0 M used swap
         2999 M free swap
      1709688 non-nice user cpu ticks
            0 nice user cpu ticks
       978557 system cpu ticks
     83005044 idle cpu ticks
       300460 IO-wait cpu ticks
          451 IRQ cpu ticks
        68838 softirq cpu ticks
            0 stolen cpu ticks
      2254405 pages paged in
    279842045 pages paged out
            0 pages swapped in
            0 pages swapped out
      7311870 interrupts
     17116315 CPU context switches
   1287559482 boot time
       320876 forks

ulimit -a :
    core file size          (blocks, -c) 0
    data seg size           (kbytes, -d) unlimited
    scheduling priority             (-e) 0
    file size               (blocks, -f) unlimited
    pending signals                 (-i) 141312
    max locked memory       (kbytes, -l) 32
    max memory size         (kbytes, -m) unlimited
    open files                      (-n) 1024
    pipe size            (512 bytes, -p) 8
    POSIX message queues     (bytes, -q) 819200
    real-time priority              (-r) 0
    stack size              (kbytes, -s) 8192
    cpu time               (seconds, -t) unlimited
    max user processes              (-u) 141312
    virtual memory          (kbytes, -v) unlimited
    file locks                      (-x) unlimited

Here are some details on memory status (in Mb) when the MySQL Server is stopping (at 1:45 AM) and restarting (at 2:15 AM) the last day it crashed :

time        memtotal    memfree     cached      active      inactive    swaptotal   swapfree    buffers
01:44:04    16072       428,9       1729,5      2599,5      12501,6     3000        3000        1084,6
01:45:04    16072       485,4       1729,5      2543,1      12500,8     3000        3000        1084,6
01:46:04    16072       1935,5      1729,5      1097,5      12500,7     3000        3000        1084,6
...
02:14:04    16072       1934,9      1729,6      1097,6      12500,8     3000        3000        1084,6
02:15:04    16072       1893,7      1729,6      1139,9      12500,5     3000        3000        1084,6
02:16:04    16072       1963,5      1659,2      1140,3      12429,8     3000        3000        1084,6

Some interesting parameters from our my.cnf :

innodb_log_file_size            = 512M
innodb_buffer_pool_size         = 8G
innodb_additional_mem_pool_size = 20M
innodb_flush_method             = O_DIRECT

We have many others servers working fine with the same configuration except :
* innodb_flush_method is not set on other servers
* this one is the only one on which replication is enabled

We searched all around the web and find a post which could be similar to our issue, but this one is quite old (2005) and no solution was given :

http://www.mail-archive.com/mysql@lists.mysql.com/msg116102.html

Any clues is welcome.

Best regards,

Fabien L.

How to repeat:
Unrepeatable : random crash...

Thank you for the report.

But version 5.1.37 very old and many bugs were fixed since. Please upgrade to current version 5.1.51 and if problem still exists send us full error log file.

Thank you for your reply.

But we cannot freely upgrade MySQL version on our production servers.

We glanced through all MySQL version from 5.1.37 to 5.1.52, but we didn't find any bug similar to the one we've described. Could you tell me which bug you were thinking of ?

Thank you for the feedback.

It is hard to say what is wrong having information provided, but as we don't backport bugfixes anyway there is a sense to upgrade to be sure problem is repeatable in newer version. Also send us full error log file.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".