Bug #13335 | running out of threads at just over 1000 connections | ||
---|---|---|---|
Submitted: | 20 Sep 2005 5:26 | Modified: | 10 Mar 2006 12:34 |
Reporter: | Robin McMillon | Email Updates: | |
Status: | No Feedback | Impact on me: | |
Category: | MySQL Server | Severity: | S2 (Serious) |
Version: | 4.1.14 | OS: | Linux (RedHat Enterprise Linux AS 3) |
Assigned to: | CPU Architecture: | Any |
[20 Sep 2005 5:26]
Robin McMillon
[20 Sep 2005 17:15]
Hartmut Holzgraefe
You are most likely hiting a ulimit restriction (eg. # of open files or # of processes), can you please add "ulimit -a" output so that we can check this?
[21 Sep 2005 15:40]
Robin McMillon
ulimit -a as root (runs mysqhd_safe): -------------------------------------------- core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) 4 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 7168 virtual memory (kbytes, -v) unlimited --------------------------------------------- as mysql (runs mysqld): --------------------------------------------- core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) 4 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 7168 virtual memory (kbytes, -v) unlimited --------------------------------------------- However, in mysqld_safe, the following code is run: --------------------------------------------- USER_OPTION="" if test -w / -o "$USER" = "root" then if test "$user" != "root" -o $SET_USER = 1 then USER_OPTION="--user=$user" fi # If we are root, change the err log to the right user. touch $err_log; chown $user $err_log if test -n "$open_files" then ulimit -n $open_files args="--open-files-limit=$open_files $args" fi if test -n "$core_file_size" then ulimit -c $core_file_size fi fi --------------------------------------------- From our my.cnf file: [mysqld_safe] open-files-limit = 8192 and a 'SHOW VARIABLES;' confirms that this is the setting in mysql. Therefore, I do not understand why we would be running up against the maximum number of open files (8192) or the maximum number of open processes (7168).
[25 Oct 2005 8:30]
Hartmut Holzgraefe
what kind of mysql package are you using? static, dynamic, built by mysql, built by redhat, compiled yourself? the linuxthreads thread library has a hard limit of 1024 threads per process by default, in our own static binaries we have patched this up to 4096, looks as if you are hitting this hard limit
[26 Nov 2005 0:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[28 Nov 2005 16:41]
Robin McMillon
To answer your questions (again), here is the information provided in my original post: >what kind of mysql package are you using? >static, dynamic, built by mysql, built by redhat, compiled yourself? static built by MySQL AB version 4.1.14 using the version built by MySQL AB with the patched Linux Threads library that you and MySQL's website claim allows up to 4096 connections Are you claiming that even though I am using *your* (MySQL's) patched version, I am still hitting the limit in the system Linux Threads?
[28 Nov 2005 16:49]
Robin McMillon
Sorry, apparently I had two accounts: To answer your questions (again), here is the information provided in my original post: >what kind of mysql package are you using? >static, dynamic, built by mysql, built by redhat, compiled yourself? static built by MySQL AB version 4.1.14 using the version built by MySQL AB with the patched Linux Threads library that you and MySQL's website claim allows up to 4096 connections Are you claiming that even though I am using *your* (MySQL's) patched version, I am still hitting the limit in the system Linux Threads?
[10 Feb 2006 12:34]
Valeriy Kravchuk
Just want to clarify current status of this report. If you still suffer from this problem, please, send the top command results (and SHOW PROCESSLIST results) when it will happen next time. Please, try to use newer version, 4.1.18 (our static build) also.
[11 Mar 2006 0:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[15 Jun 2006 1:15]
Andrew Jones
I was having this same issue and have solved it by raising the hard limit of open files in /etc/security/limits.conf Even if you raise the value of open files in the startup scripts, It still cannot go above the hard limit, which is 1024 by default. /etc/security/limits.conf: mysql soft nofile 4096 mysql hard nofile 4096 This seems not to be a mysql bug, but rather OS imposed limits. I wrote about it here: http://www.haqthegibson.com/article/34 Cheers, Jonesy
[15 Jun 2006 1:17]
Andrew Jones
Sorry, that snippet of /etc/security/limits.conf in my last comment should have been more clearly described, that is what I added to increase the limits. Btw, that was on RedHat Enterprise Linux but applies to other Linux as well. Jonesy
[22 Mar 2010 4:24]
Mark Garcia
I had this problem before .... I've set key_buffer_size to something lower like between 8 to 64M and set thread_stack = 356k and that worked for me .... Hope this helps