Bug #34084 mysqld_safe restarts mysqld after a crash when /etc/init.d/mysql stop is run
Submitted: 27 Jan 2008 3:13 Modified: 12 Apr 2013 17:35
Reporter: Mark Callaghan Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Packaging Severity:S1 (Critical)
Version:5.0.37, 5.1.61 OS:Any (Linux)
Assigned to: Assigned Account CPU Architecture:Any
Tags: Contribution, crash, mysqld_safe, shutdown

[27 Jan 2008 3:13] Mark Callaghan
Description:
If mysqld crashes during shutdown when done by /etc/init.d/mysql stop, then mysqld_safe restarts mysqld. Tools that expect mysqld to stop after running /etc/init.d/mysql stop then get stuck.

We get intermittent crashes on shutdown. See http://bugs.mysql.com/?id=34045. 

How to repeat:
described above

Suggested fix:
Don't restart mysqld if it crashes during shutdown.
[29 Jan 2008 13:26] Susanne Ebrecht
Many thanks for writing a bug report.

MySQL 5.0.37 is quiet old. At the moment our newest version is MySQL 5.0.51 (available as source)

To analyze your problem, we need to know:

1) the exact version of your operating system
2) how did you install MySQL? Which packages do you use?
[29 Jan 2008 16:26] Mark Callaghan
The behavior occurs because mysqld removes its PID file on shutdown. If it crashes during shutdown before it reaches the code to remove the PID file, the file does not get removed (grep for pidfile_name in mysqld.cc).

mysqld_safe restarts mysqld when the PID file has not been removed:
  if test ! -f $pid_file                # This is removed if normal shutdown
  then
    echo "STOPPING server from pid file $pid_file"
    break
  fi
[30 Jan 2008 16:33] Susanne Ebrecht
Thank you for taking the time to report a problem.  Unfortunately you are not using a current version of the product you reported a problem with -- the problem might already be fixed. Please download a new version from http://www.mysql.com/downloads/

Our newest version at the moment is: MySQL 5.0.51a

If you are able to reproduce the bug with one of the latest versions, please change the version on this bug report to the version you tested and change the status back to "Open".  Again, thank you for your continued support of MySQL.
[30 Jan 2008 19:02] Mark Callaghan
I am disappointed by this response. Is the new support policy to mark as unsupported any bug reported against a release prior to the latest community release? If so, you should publicize it, but I think the community response to it will be very negative.

Are you suggesting that the behavior has changed for this from 5.0.37 to 5.0.51 or just guessing? This should be easy for you to check.
[30 Jan 2008 19:27] MySQL Verification Team
Re-opened to check against the latest source.
[30 Jan 2008 22:12] James Day
This is known behavior since 4.0 or earlier. Has been discussed before, suggest checking with Tim to see if he's looked at it.
[31 Jan 2008 10:19] Sveta Smirnova
Thank you for the report.

Verified as described.

To repeat:

1. Patch sql/mysqld.cc:

$diff -u sql/mysqld.cc sql/mysqld.cc2
--- sql/mysqld.cc       2008-01-31 11:11:50.000000000 +0100
+++ sql/mysqld.cc2      2008-01-31 11:11:41.000000000 +0100
@@ -3761,6 +3761,7 @@
       CloseHandle(hEventShutdown);
   }
 #endif
+sleep(20);
   wait_for_signal_thread_to_end();
   clean_up_mutexes();
   my_end(opt_endinfo ? MY_CHECK_ERROR | MY_GIVE_INFO : 0);

2. Start mysqld with help of mysqld_safe
3. issue mysql.server stop
4. Immediately run `kill -9 mysqld_pid` in another shell session
5. See result

Workaround: as root `kill -9 mysqld_safe`
[31 Jan 2008 11:24] Sveta Smirnova
Problem is with mysqld_safe, not with mysql.server script.

One can replace instruction

3. issue mysql.server stop

with

3. issue mysqladmin shutdown

in "How to repeat" section as well
[1 Feb 2008 8:56] Sveta Smirnova
Patch for mysqld_safe to allow it die after repeatable mysqld restarts:

$diff -u ~/bk/mysql-5.0/scripts/mysqld_safe /Users/sveta/Applications/utils/mysql/bin/mysqld_safe_patched.sh
--- /Users/sveta/bk/mysql-5.0/scripts/mysqld_safe       2008-02-01 01:36:54.000000000 +0300
+++ /Users/sveta/Applications/utils/mysql/bin/mysqld_safe_patched.sh    2008-02-01 11:52:34.000000000 +0300
@@ -12,6 +12,12 @@
 
 KILL_MYSQLD=1;
 MYSQLD=
+DIE_ON_REPEATABLE_CRASH=0;
+RETRIES=10;
+TIMEOUT=60;
+RUN_ON_DEATH=
+CRASHES=0;
+RESTART_TIME=`date "+%s"`
 
 trap '' 1 2 3 15                       # we shouldn't let anyone kill us
 
@@ -39,6 +45,10 @@
   --mysqld-version=VERSION   Use "mysqld-VERSION" as mysqld
   --nice=NICE                Set the scheduling priority of mysqld
   --skip-kill-mysqld         Don't try to kill stray mysqld processes
+  --die-on-repeatable-crash  Die in case of repeatable mysqld crashes
+  --retries[=10]             Retries before die
+  --timeout[=60]             Timeout between retries (in seconds)
+  --run-on-death=COMMAND     Run COMMAND on death
 
 All other options are passed to the mysqld program.
 
@@ -94,6 +104,10 @@
       --help)
         usage
         ;;
+         --die-on-repeatable-crash*) DIE_ON_REPEATABLE_CRASH=1; ;;
+         --retries=*) RETRIES=`echo "$arg" | sed -e "s;--retries=;;"` ;;
+         --timeout=*) TIMEOUT=`echo "$arg" | sed -e "s;--timeout=;;"` ;;
+         --run-on-death=*) RUN_ON_DEATH=`echo "$arg" | sed -e "s;--run-on-death=;;"` ;;
       *)
         if test -n "$pick_args"
         then
@@ -106,7 +120,6 @@
   done
 }
 
-
 #
 # First, try to find BASEDIR and ledir (where mysqld is)
 # 
@@ -422,6 +435,20 @@
       I=`expr $I + 1`
     done
   fi
+  if [[ $DIE_ON_REPEATABLE_CRASH ]]
+  then
+         if (( $TIMEOUT < (( - $RESTART_TIME + (RESTART_TIME=`date "+%s"`) )) ))
+         then
+         echo "in";
+               CRASHES=0;
+         fi
+         (( CRASHES++ ))
+         if (( $RETRIES < $CRASHES ))
+         then
+               exec $RUN_ON_DEATH
+               exit 0;
+         fi
+  fi
   echo "`date +'%y%m%d %H:%M:%S'`  mysqld restarted" | tee -a $err_log
 done
[1 Feb 2008 9:16] Sveta Smirnova
patch for mysqld_safe.sh as well

Attachment: mysqld_safe.sh.patch (application/octet-stream, text), 1.67 KiB.

[11 Nov 2008 22:16] Sveta Smirnova
Bug #40423 was marked as duplicate of this one.
[8 Dec 2008 21:49] Sveta Smirnova
Mark,

could you please try patch posted in my comment "[1 Feb 9:56] Sveta Smirnova" and inform us if it solves your problem? Thanks in advance.
[9 Jan 2009 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[9 Jan 2009 18:01] Mark Callaghan
It is on my list of things to do.
[30 Mar 2009 17:49] Mark Callaghan
It is still on my list of things to do. I don't have time to provide a better response.
[18 Aug 2010 18:02] Shlomo Priymak
We have encountered the same problem when using a redhat clustering solution with mysql.
The cluster tried to shut down MySQL which crashed to perform a failover. But, since mysqld_safe is restarting it - the cluster gets confused and we get a split-brain scenario.

The part of this script that is relevant for us was disabling the auto-restart feature, but the other fixes there seem great as well. 
The script is outdated, and it's time for it to get an overhaul.
Can you please officially release this patch so that it would be a part of an upcoming next version?
[18 Mar 2012 18:39] Valeriy Kravchuk
This is still a verified bug. Sveta's patch is not applied to any current branch, as far as I can see.
[12 Apr 2013 17:35] Paul Dubois
Noted in 5.7.2 changelogs.

If mysqld crashed during a shutdown initiated by /etc/init.d/mysql
stop, mysqld_safe restartd mysqld when it should not have.