Bug #14906 MySQL exit itself
Submitted: 13 Nov 2005 6:00 Modified: 22 Jul 2008 16:09
Reporter: Kam Kam Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:5.0.15 OS:Any (Trustix 3)
Assigned to: CPU Architecture:Any

[13 Nov 2005 6:00] Kam Kam
Description:
I am using MySQL5 (mysql-standard-5.0.15-linux-i686, directly download from mysql.com), the MySQL service exit itself and I have found the error log message in error log. My platform is running Trustix 3.0 (kernel 2.6.12.6) on Dell 1850 (Dual Xeon 3GHz, 2GB DDR2 RAM)

How to repeat:
the error log message how me that the mysql server keep restart itself and sometimes 'Out of memory':

-----------
Number of processes running now: 0
051111 19:17:41  mysqld restarted
/usr/local/mysql5/bin/mysqld: File './db-slow.log' not found (Errcode: 13)
051111 19:17:42 [ERROR] Could not use db-slow.log for logging (error 13). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 357273600 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 267955200 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 200964096 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 150721536 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 113040384 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 84777984 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 63581184 bytes)
051111 19:17:46 [Note] /usr/local/mysql5/bin/mysqld: ready for connections.
Version: '5.0.15-standard-log'  socket: '/var/run/mysql/mysql5.sock'  port: 53306  MySQL Community Edition - Standard (GPL)

Number of processes running now: 0
051111 19:32:09  mysqld restarted
/usr/local/mysql5/bin/mysqld: File './db-slow.log' not found (Errcode: 13)
051111 19:32:11 [ERROR] Could not use db-slow.log for logging (error 13). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 357273600 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 267955200 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 200964096 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 150721536 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 113040384 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 84777984 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 63581184 bytes)
051111 19:32:14 [Note] /usr/local/mysql5/bin/mysqld: ready for connections.
Version: '5.0.15-standard-log'  socket: '/var/run/mysql/mysql5.sock'  port: 53306  MySQL Community Edition - Standard (GPL)

Number of processes running now: 0
051111 19:47:10  mysqld restarted
/usr/local/mysql5/bin/mysqld: File './db-slow.log' not found (Errcode: 13)
051111 19:47:14 [ERROR] Could not use db-slow.log for logging (error 13). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 357273600 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 267955200 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 200964096 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 150721536 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 113040384 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 84777984 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 63581184 bytes)
051111 19:47:22 [Note] /usr/local/mysql5/bin/mysqld: ready for connections.
Version: '5.0.15-standard-log'  socket: '/var/run/mysql/mysql5.sock'  port: 53306  MySQL Community Edition - Standard (GPL)
051111 20:09:30 [Warning] Aborted connection 3 to db: 'xx' user: 'xx' host: 'localhost' (Got timeout reading communication packets)

Number of processes running now: 0
051111 21:40:05  mysqld restarted
/usr/local/mysql5/bin/mysqld: File './db-slow.log' not found (Errcode: 13)
051111 21:40:06 [ERROR] Could not use db-slow.log for logging (error 13). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 357273600 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 267955200 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 200964096 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 150721536 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 113040384 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 84777984 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 63581184 bytes)
051111 21:40:17 [Note] /usr/local/mysql5/bin/mysqld: ready for connections.
Version: '5.0.15-standard-log'  socket: '/var/run/mysql/mysql5.sock'  port: 53306  MySQL Community Edition - Standard (GPL)

Number of processes running now: 0
051111 22:33:49  mysqld restarted
/usr/local/mysql5/bin/mysqld: File './db-slow.log' not found (Errcode: 13)
051111 22:33:53 [ERROR] Could not use db-slow.log for logging (error 13). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 357273600 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 267955200 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 200964096 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 150721536 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 113040384 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 84777984 bytes)
/usr/local/mysql5/bin/mysqld: Out of memory (Needed 63581184 bytes)
051111 22:34:17  mysqld ended
[13 Nov 2005 10:50] Valeriy Kravchuk
Thank you for a problem report. Please, send the my.cnf content, the df -k command results, the ulimit -a command results performed by the used who runs mysqld,  and describe what actions are performed on your MySQL server when it get restarted.
[15 Nov 2005 12:34] Kam Kam
while the mysql restarted itself, i just running some insert / select query

-----------------------------------------------------------------------
::my.cnf

[client]
port = 3306
socket = /tmp/mysql.sock

[mysqld]
tmpdir = /tmp/mysql
port = 53306
socket = /var/run/mysql/mysql5.sock
log-warnings
log-slow-queries
skip-locking
skip-innodb
skip-bdb
key_buffer = 384M
max_allowed_packet = 1M
table_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache = 8
query_cache_size = 32M
thread_concurrency = 8
server-id = 213

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

-----------------------------------------------------------------------
::df -k

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda4             28067892   2952296  25115596  11% /
/dev/sda1               256666     20388    223026   9% /boot
/dev/sda2              5116508     32840   5083668   1% /tmp

-----------------------------------------------------------------------
::ulimit -a ./bin/mysqld_safe --user=mysql

core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) 32
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 16382
virtual memory        (kbytes, -v) unlimited
[15 Nov 2005 12:59] Valeriy Kravchuk
Thank you for the additional information. How many concurrect connections do you have? With default max_connections = 100, you may have 384 + (2+2)*100 + 32 = 800+ M of RAM used for MySQL server only. Are there any other resource-intensive processes running at the same time?

Please, send the results of the "top" Linux command executed under the peak load of your server, as well as the result of SHOW PROCESSLIST statement executed at the same time.
[16 Nov 2005 6:02] Kam Kam
just running mysql and a few php script working on mysql.

------------------------------------------------------------------------------------
:: top
top - 13:58:59 up 41 days,  1:19,  1 user,  load average: 0.37, 0.13, 0.04
Tasks:  95 total,   1 running,  94 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.7% us,  0.2% sy,  0.0% ni, 98.2% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   2076136k total,   286296k used,  1789840k free,      336k buffers
Swap:  2096472k total,    64872k used,  2031600k free,   138696k cached
[16 Nov 2005 18:31] Valeriy Kravchuk
OK, so, it looks like you have more that enough memory for your queries. Time to see them. Can you identify one or several SQL statements that leads to the problem? You may use the SHOW PROCESSLIST command being executed for some time in parallel with you php scripts, or just add some logging to them. Or start MySQL with general query log.

Please, try to identifdy the problematic statements and send them.
[17 Dec 2005 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[31 Aug 2007 18:28] Michal Mier
I had the same error about slow queries logging file. Maybe it is a bug with resolving filepath string of log-slow-queries variable, because setting the value just to a file name solved the problem.
So:
log-slow-queries = /var/log/mysql.slow.log # Does not work!
log-slow-queries = mysql.slow.log # Works! MySQL takes it from {datadir} directory

------------------------------------------------------
MYSQL:
protocol_version         10                                                        
version                  5.0.27-log                                                
version_bdb              Sleepycat Software: Berkeley DB 4.1.24: (October 21, 2006)
version_comment          Source distribution                                       
version_compile_machine  i686                                                      
version_compile_os       redhat-linux-gnu 
------------------------------------------------------
[26 May 2008 9:47] Mattias Malmgren
Got the same problem, but I don't get any out of memmory problems. Here is more information:
http://sh.freefarm.se/mysql/

Help?!
[22 Jun 2008 16:09] Valeriy Kravchuk
Mattias,

I have problems getting your error log:

hastavelsforbundet.err.gz (uncompress with gunzip) - this archive is broken
hastavelsforbundet.err (not compressed at all) - forbidden

Please, upload your error log to this bug report if you think that your situation is similar.
[22 Jul 2008 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".