Description:
mysqld hangs when it's shut down and .pid file is located at a non-writable dir. The log message says: Shutdown completed. So it's not very harmful. But it never returns unless pressing Ctrl+C. A message saying how long have been waiting is displayed: Warning; Aborted waiting on pid file: '/usr/local/mysql/test.pid' after XX seconds.
How to repeat:
Suppose mysql is install in /usr/local/mysql (i.e. basedir=/usr/local/mysql). This directory is owned by and grouped in root according to the manual. Make sure this folder is not writable for user mysql.
Suppose we change the configuration option to be
pid-file=/usr/local/mysql/test.pid
and create a test.pid file there:
touch /usr/local/mysql/test.pid
sudo chown mysql test.pid
sudo chgrp mysql test.pid
Then run mysqld:
sudo /usr/local/mysql/bin/mysqld --user=mysql&
Then shut down mysqld:
sudo /usr/local/mysql/bin/mysqladmin shutdown
After showing "[Note] /usr/local/mysql/bin/mysqld: Shutdown complete"
it never returns.
Pressing Ctrl+C shows:
"Warning; Aborted waiting on pid file: '/usr/local/mysql/test.pid' after XX seconds"
Suggested fix:
The culprit is that mysqld can write (delete) a preexisting file test.pid but is unable to update the directory /usr/local/mysql's inode entry. Not sure about the detail implementation. The OS should return a value signaling the failure of writing /usr/local/mysql's inode. mysqld should have reported this failure and then stopped instead of hanging (seems it's trying to write indefinitely).
This problem also applies to the socket option. But mysqld_safe may avoid this problem.
I guess this could have been improved by
(1) Each time discovering an existing files for the pid-file or socket options (by default XXX.pid and /tmp/mysql.sock), mysqld should sound an alarm in the log because these files are supposed to existent only when mysqld is running. Something suspicious might have happened.
(2) If (1) occurs. mysqld should check whether it's able to delete a specific file (or can update that residing directory's inode or not). If not, just abort with an error message.
(3) An indefinite loop to try to write should be avoided. If this is the case in implementing the final deletion of .pid file (or .sock file), this should abort immediately a failure is returned from the OS.
Though the case is a little bit contrived and the hang is somewhat harmless (shutdown is completed), it's possible to occur in real-world practice and may bother users.