Bug #32912 /usr/sbin/mysqld: Out of memory
Submitted: 2 Dec 2007 17:08 Modified: 4 Jan 2008 10:54
Reporter: hanji hanj Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:5.0.44-r2 OS:Linux (Gentoo)
Assigned to: CPU Architecture:Any
Tags: error, gentoo, linux, mysqld, out of memory

[2 Dec 2007 17:08] hanji hanj 
Description:
I'm receiving intermittent "Out of memory" errors. When this occurs all web sites are down, but other services depending on mysql run fine (ie: postfix, courier, cacti, cron based apps -- PHP/CLI). Only way to recover is to restart mysql. It appears that mysql can run fine without incidence for many days, in fact the last crash was almost a month ago.

Here is the error message in mysqld.err

071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:15:03 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 257408 bytes)
071202  1:16:51 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 256008 bytes)

Here is a output from top during problem phase, including mysql process

top - 07:19:33 up 33 days, 23:21,  3 users,  load average: 0.24, 0.27, 0.28
Tasks:  99 total,   1 running,  98 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.4%us,  0.5%sy,  0.0%ni, 95.3%id,  0.7%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   1540848k total,  1505512k used,    35336k free,   314200k buffers
Swap:   976744k total,      376k used,   976368k free,   411056k cached

1212 mysql     15   0 4095m  56m 4828 S  0.0  3.8  30:30.85 mysqld

Here is my meminfo output:

MemTotal:      1540848 kB
MemFree:         37360 kB
Buffers:        310400 kB
Cached:         425496 kB
SwapCached:         36 kB
Active:         852720 kB
Inactive:       389072 kB
HighTotal:      638912 kB
HighFree:         1168 kB
LowTotal:       901936 kB
LowFree:         36192 kB
SwapTotal:      976744 kB
SwapFree:       976708 kB
Dirty:             180 kB
Writeback:           0 kB
AnonPages:      505912 kB
Mapped:          25736 kB
Slab:           253712 kB
SReclaimable:   242696 kB
SUnreclaim:      11016 kB
PageTables:       2236 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   1747168 kB
Committed_AS:  1047296 kB
VmallocTotal:   114680 kB
VmallocUsed:       652 kB
VmallocChunk:   113928 kB

My current kernel is 2.6.22-hardened-r8 (hardened-sources). My distro is Gentoo Linux.

This problem occured with 5.0.44, and 5.0.44-r2 (it did not occur with 5.0.44-r1, but that version was quickly followed up with r2)

How to repeat:
Unable to repeat. Intermittent problem.

Suggested fix:
Restart of mysql service.
[2 Dec 2007 20:27] Valeriy Kravchuk 
Thank you for a problem report. Please, send the results of:

uname -a
ulimit -a
file `which mysqld`
[2 Dec 2007 21:39] hanji hanj 
Thanks.. here ya go...

uname -a
Linux removed.domain.com 2.6.22-hardened-r8 #1 Mon Oct 29 08:46:42 MDT 2007 i686 AMD Sempron(tm) 2600+

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 12159
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 12159
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

file `which mysqld`
/usr/sbin/mysqld: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), stripped
[4 Dec 2007 10:54] Valeriy Kravchuk 
OK, so it is 32-bit OS with 32-bit MySQL server. When server tried to use almost 4G of memory (for whatever reason. like a lot of MEMORY and/or temporary tables used) it fails with "Out of memory" error message. This is expected. Why do you think there is any bug here?
[5 Jan 2008 0:00] Bugs System 
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".