Bug #44616 open_files_limit limits
Submitted: 1 May 2009 19:28 Modified: 28 Feb 2011 19:54
Reporter: Gatis Rumbens Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Options Severity:S5 (Performance)
Version:5.1.30 OS:FreeBSD (7.1 AMD64)
Assigned to: CPU Architecture:Any
Tags: open_files_limit, Too many open files

[1 May 2009 19:28] Gatis Rumbens
Description:
I have heavy loaded MySQL DB server (FreeBSD 7.1 AMD64 with MySQL 5.1.30) ~1000-1500req/sec (Hardware: 4 x Quad cores, 32G ram, 6raid...) The main problem is that almost all time i got errors like 22,23,24 (Too many open files etc)...there is no problem with cpu/ram performance to handle all this requests but main problem is with this files descriptor limit.

The main used table is MyISAM with LIST partitioning (128 partitions).

In the OS kernel this values are:

kern.maxfiles=131072(2x more than mysql variable)
kern.maxfilesperproc=131072(2x more than mysql variable)

# fstat | grep mysql | wc -l
   64581

So as you can see this value is very very closely to MySQL variable:

open_files_limit = 65535 (This is hard coded maximum..but why?)

I've decreased max_connections and table_cache variables to unequipped open files. But in this case I'm getting errors: Too many connections...

Is it possible some how increase 
open_files_limit variable more than 65535 ?

How to repeat:
-

Suggested fix:
Allow increase open_files_limit variable.
[3 Jun 2009 9:33] Sveta Smirnova
Thank you for the report.

You should be able to increase open-files-limit if OS allows you to do so. Please provide output of `ulimit -a` for user you run mysqld as.
[3 Jul 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[3 Aug 2010 8:28] Samuel Streiner
I've got the same Problem with FreeBSD 7.2

ulimit -a output:
-----------------
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) 524288
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 11095
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 65536
cpu time               (seconds, -t) unlimited
max user processes              (-u) 5547
virtual memory          (kbytes, -v) unlimited

The Bug only occurs occasional. The Server does pretty much nothing at all. If the Bug once occurs, I have to restart mysqld.
[3 Aug 2010 8:54] Sveta Smirnova
Samuel,

thank you for the feedback. Which exact version of MySQL do you use? Does FLUSH TABLES help to solve the problem? Please send us your configuration file and output of SHOW GLOBAL STATUS LIKE '%open%' in time when error occurs.
[5 Aug 2010 10:46] Samuel Streiner
Hi Sveta!

Here is my my.cnf
--------------------------------------------------------------------
# Example MySQL config file for medium systems.
#
# This is for a system with little memory (32M - 64M) where MySQL plays
# an important part, or systems up to 128M where MySQL is used together with
# other programs (such as a web server)
#
# You can copy this file to
# /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options (in this
# installation this directory is /var/db/mysql) or
# ~/.my.cnf to set user-specific options.
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.

# The following options will be passed to all MySQL clients
[client]
#password       = your_password
port            = 3306
socket          = /tmp/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port            = 3306
socket          = /tmp/mysql.sock
skip-locking
key_buffer = 16M
max_allowed_packet = 16M
table_cache = 64
sort_buffer_size = 2M
net_buffer_length = 16K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
# 
#skip-networking

# Replication Master Server (default)
# binary logging is required for replication
#log-bin
#binlog-do-db = cms_atsi_dev

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
#server-id = 1

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
#    the syntax is:
#
#    CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
#    MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
#    where you replace <host>, <user>, <password> by quoted strings and
#    <port> by the master's port number (3306 by default).
#
#    Example:
#
#    CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
#    MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
#    start replication for the first time (even unsuccessfully, for example
#    if you mistyped the password in master-password and the slave fails to
#    connect), the slave will create a master.info file, and any later
#    change in this file to the variables' values below will be ignored and
#    overridden by the content of the master.info file, unless you shutdown
#    the slave server, delete master.info and restart the slaver server.
#    For that reason, you may want to leave the lines below untouched
#    (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id       = 2
#
# The replication master for this slave - required
#master-host     =   <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user     =   <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password =   <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port     =  <port>
#
# binary logging - not required for slaves, but recommended
#log-bin

# Point the following paths to different dedicated disks
#tmpdir         = /tmp/         
#log-update     = /path-to-dedicated-directory/hostname

# Uncomment the following if you are using BDB tables
#bdb_cache_size = 4M
#bdb_max_lock = 10000

# Uncomment the following if you are using InnoDB tables
innodb_data_home_dir = /var/db/mysql/
innodb_data_file_path = ibdata1:16M:autoextend:max:1024M
innodb_file_per_table = 1
#innodb_force_recovery = 3
#innodb_log_group_home_dir = /var/db/mysql/
#innodb_log_arch_dir = /var/db/mysql/
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 5M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout
--------------------------------------------------------------------

I'll provide the other infos, as soon as the problem reoccurs!
[5 Aug 2010 10:49] Samuel Streiner
SELECT Version(); => 5.1.30
[5 Aug 2010 10:55] Samuel Streiner
This is the only machine where this error occurs! My other machines are running FreeBSD 8 with MySQL 5.5.4-m3. And the are all running fine! So maybe it's just some really stange race contition concerning FreeBSD 7.x + MySQL 5.1.30...

Plus I have to mention that MySQL runs in a Jail! :)
[7 Aug 2010 12:56] Sveta Smirnova
Thank you for the feedback.

Cache-related variable like table_cache are low enough, so should not affect this.

If you don't see this error with newer versions please try current 5.1 version which is 5.1.49 on problem box and if problem still exists inform us if FLUSH TABLES solves the problem.
[7 Sep 2010 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[26 Jan 2011 19:12] Samuel Streiner
And again!

But this time I was lucky enough to get some more informations! I did a lsof for my mysql-process... And it shows me about 11099 open files. And that exceeds my systemsetting for kern.maxfilesperproc (sysctl kern.maxfiles=20000, sysctl kern.maxfilesperproc=11095).

One thing thats funny: 10730 of them are actually UDP connections...
Like: mysqld  20199 mysql *953u  IPv4 0xca27c438                  0t0      UDP server.example.com:51431

Quite interesting...
[28 Jan 2011 19:54] Sveta Smirnova
Thank you for the feedback.

Looks strange: there should not be UDP connections. Is it possible what you experience some kind of attack on MySQL server?

Please also upgrade to current version 5.1.54, try with it and inform us if problem still exists.
[1 Mar 2011 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".