Bug #64643 mysqld hangs when it's shut down and .pid file is located at a non-writable dir
Submitted: 14 Mar 2012 4:25 Modified: 20 Apr 2012 0:13
Reporter: Dongcai Shen Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server: Options Severity:S3 (Non-critical)
Version:5.5.21 OS:Linux
Assigned to: CPU Architecture:Any

[14 Mar 2012 4:25] Dongcai Shen
Description:
mysqld hangs when it's shut down and .pid file is located at a non-writable dir. The log message says: Shutdown completed. So it's not very harmful. But it never returns unless pressing Ctrl+C. A message saying how long have been waiting is displayed: Warning;  Aborted waiting on pid file: '/usr/local/mysql/test.pid' after XX seconds.

How to repeat:
Suppose mysql is install in /usr/local/mysql (i.e. basedir=/usr/local/mysql). This directory is owned by and grouped in root according to the manual. Make sure this folder is not writable for user mysql. 

Suppose we change the configuration option to be 
pid-file=/usr/local/mysql/test.pid 

and create a test.pid file there:
touch /usr/local/mysql/test.pid
sudo chown mysql test.pid
sudo chgrp mysql test.pid

Then run mysqld:
sudo /usr/local/mysql/bin/mysqld --user=mysql&

Then shut down mysqld:
sudo /usr/local/mysql/bin/mysqladmin shutdown

After showing "[Note] /usr/local/mysql/bin/mysqld: Shutdown complete"
it never returns.
Pressing Ctrl+C shows:
"Warning;  Aborted waiting on pid file: '/usr/local/mysql/test.pid' after XX seconds"

Suggested fix:
The culprit is that mysqld can write (delete) a preexisting file test.pid but is unable to update the directory /usr/local/mysql's inode entry. Not sure about the detail implementation. The OS should return a value signaling the failure of writing /usr/local/mysql's inode. mysqld should have reported this failure and then stopped instead of hanging (seems it's trying to write indefinitely).

This problem also applies to the socket option. But mysqld_safe may avoid this problem.

I guess this could have been improved by 

(1) Each time discovering an existing files for the pid-file or socket options (by default XXX.pid and /tmp/mysql.sock), mysqld should sound an alarm in the log because these files are supposed to existent only when mysqld is running. Something suspicious might have happened.

(2) If (1) occurs. mysqld should check whether it's able to delete a specific file (or can update that residing directory's inode or not). If not, just abort with an error message.

(3) An indefinite loop to try to write should be avoided. If this is the case in implementing the final deletion of .pid file (or .sock file), this should abort immediately a failure is returned from the OS.

Though the case is a little bit contrived and the hang is somewhat harmless (shutdown is completed), it's possible to occur in real-world practice and may bother users.
[20 Apr 2012 0:13] Sveta Smirnova
Thank you for the report.

But this is not a bug. You just did not wait long enough: after 3600 seconds mysqladmin prints "Warning;  Aborted waiting on pid file: './tmp/m.pid' after 3600 seconds", then exists. And mysqld process really stops.