MySQL Bugs: #56372: fd leak in mysql / innodb in use

Bug #56372	fd leak in mysql / innodb in use
Submitted:	30 Aug 2010 14:43	Modified:	30 Apr 2012 12:14
Reporter:	Quentin Arce	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S3 (Non-critical)
Version:	5.1.42-community-log	OS:	Linux (REHEL 5.2)
Assigned to:		CPU Architecture:	Any

Description:
I found one other bug which noted an FD leak in version 5.0.x but, it also had a note about a memory and the FD issue seemed to be skipped over.

I have an mysqld which is using innodb.  I just loaded a dump of a production db which is 50 GB into this new server and noticed the following in the fd list for mysql.

ls -l /proc/32736/fd/ | grep dele
lrwx------ 1 root root 64 Aug 30 07:23 12 -> /tmp/ibzwGkjh (deleted)
lrwx------ 1 root root 64 Aug 30 07:23 5 -> /tmp/ibObcihH (deleted)
lrwx------ 1 root root 64 Aug 30 07:23 6 -> /tmp/ibur7OaA (deleted)
lrwx------ 1 root root 64 Aug 30 07:23 7 -> /tmp/ibAY8l4s (deleted)
lrwx------ 1 root root 64 Aug 30 07:23 8 -> /tmp/ibVB8TOm (deleted)

Looking at the production server there is also a number of file descriptor like the above which look like leaked fds...

ls -l /proc/32251/fd/ | grep del
lrwx------ 1 root root 64 Aug 30 07:38 12 -> /tmp/ibF1cqsH (deleted)
lrwx------ 1 root root 64 Aug 30 07:38 5 -> /tmp/ib8V4kyk (deleted)
lrwx------ 1 root root 64 Aug 30 07:38 6 -> /tmp/ibz7VvqU (deleted)
lrwx------ 1 root root 64 Aug 30 07:38 7 -> /tmp/ibNrWGiu (deleted)
lrwx------ 1 root root 64 Aug 30 07:38 8 -> /tmp/ibzWNSf4 (deleted)

As I'm filling out this form I just noticed they are the same fd numbers.  This is on two different servers.  I'm wondering if these are reused file descriptors.  In any case It seems bad to hold onto a fd when the file no longer exists.

Your call.  I don't dev on mysqld.

How to repeat:
Start up a mysqld, load a big db in and look at the proc dir of the pid for the fd list.

Thank you for the report.

Why do you think "deleted" file still exists and MySQL can use it?

>>> Why do you think "deleted" file still exists and MySQL can use it?

I think I didn't make it clear... I think the daemon is holding the FD open still in the source code.  Even though the file has been removed.

This would be something along the lines of using the glibc function

http://www.gnu.org/software/libc/manual/html_mono/libc.html#Temporary-Files

It looks like at least one process is not closing the file handles.

Perhaps (I'm going to guess here, I have not looked at the code)  A main process stared up, used a few temp files forked then the parent closed the temp files.  The child perhaps didn't close the open file handles it inherited from the parent process.  The point is at least one mysqld process is holding file descriptors open when it doesn't need to.

Thank you for the feedback.

This reminds me bug #47623 Do you actively use BLOBs?

no we do note

not... not note ;-)

all data is a string or a number or a date

I guess there's such more-or-less commonplace unix programming technique, when you create/open a file, delete it, and use resulting file descriptor as a temporary file. At the moment you close the file descriptor, the diskspace is reclaimed to OS. That is, there's no difference for OS in which point-of-time you delete the file, but there's less chance to forget if you do it at the very beginning :)

(hope that backslashes will survive)

[user@machine~]$ ( echo cockadoodledoo; sleep 5 ) > /tmp/QQ & \
  ls -l /proc/${!}/fd; \
  rm /tmp/QQ; \
  ls -l /proc/${!}/fd; \
  cat /proc/${!}/fd/1
[1] 1287
total 0
lrwx------ 1 user group 64 Feb 18 15:01 0 -> /dev/pts/7
l-wx------ 1 user group 64 Feb 18 15:01 1 -> /tmp/QQ
lrwx------ 1 user group 64 Feb 18 15:01 2 -> /dev/pts/7
lrwx------ 1 user group 64 Feb 18 15:01 255 -> /dev/pts/7
total 0                 
lrwx------ 1 user group 64 Feb 18 15:01 0 -> /dev/pts/7
l-wx------ 1 user group 64 Feb 18 15:01 1 -> /tmp/QQ (deleted)
lrwx------ 1 user group 64 Feb 18 15:01 2 -> /dev/pts/7
lrwx------ 1 user group 64 Feb 18 15:01 255 -> /dev/pts/7
cockadoodledoo
[user@machine~]$

(source: http://linsovet.com/recovery-delete-file-but-still-opened )

I doubt this is a bug. innodb creates 5 temporary files for it's own internal use at server startup.  they are deleted when mysqld shuts down.  so where is the leak?

Take a look where os_file_create_tmpfile function is called from in the source.

1. in dict_init(): dict_foreign_err_file = os_file_create_tmpfile();
2. in lock_sys_create(): lock_latest_err_file = os_file_create_tmpfile();
3. in innobase_start_or_create_for_mysql: srv_monitor_file = os_file_create_tmpfile();
4. in innobase_start_or_create_for_mysql: srv_dict_tmpfile = os_file_create_tmpfile();
5. in innobase_start_or_create_for_mysql srv_misc_tmpfile = os_file_create_tmpfile();

Its a bug because innodb will grow the file during a large transaction,  and if the disk fills up your only choice is to restart the server.