MySQL Bugs: #34084: mysqld_safe restarts mysqld after a crash when /etc/init.d/mysql stop is run

Bug #34084	mysqld_safe restarts mysqld after a crash when /etc/init.d/mysql stop is run
Submitted:	27 Jan 2008 3:13	Modified:	12 Apr 2013 17:35
Reporter:	Mark Callaghan	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Packaging	Severity:	S1 (Critical)
Version:	5.0.37, 5.1.61	OS:	Any (Linux)
Assigned to:	Assigned Account	CPU Architecture:	Any
Tags:	Contribution, crash, mysqld_safe, shutdown

Description:
If mysqld crashes during shutdown when done by /etc/init.d/mysql stop, then mysqld_safe restarts mysqld. Tools that expect mysqld to stop after running /etc/init.d/mysql stop then get stuck.

We get intermittent crashes on shutdown. See http://bugs.mysql.com/?id=34045. 

How to repeat:
described above

Suggested fix:
Don't restart mysqld if it crashes during shutdown.

Many thanks for writing a bug report.

MySQL 5.0.37 is quiet old. At the moment our newest version is MySQL 5.0.51 (available as source)

To analyze your problem, we need to know:

1) the exact version of your operating system
2) how did you install MySQL? Which packages do you use?

The behavior occurs because mysqld removes its PID file on shutdown. If it crashes during shutdown before it reaches the code to remove the PID file, the file does not get removed (grep for pidfile_name in mysqld.cc).

mysqld_safe restarts mysqld when the PID file has not been removed:
  if test ! -f $pid_file                # This is removed if normal shutdown
  then
    echo "STOPPING server from pid file $pid_file"
    break
  fi

Thank you for taking the time to report a problem.  Unfortunately you are not using a current version of the product you reported a problem with -- the problem might already be fixed. Please download a new version from http://www.mysql.com/downloads/

Our newest version at the moment is: MySQL 5.0.51a

If you are able to reproduce the bug with one of the latest versions, please change the version on this bug report to the version you tested and change the status back to "Open".  Again, thank you for your continued support of MySQL.

I am disappointed by this response. Is the new support policy to mark as unsupported any bug reported against a release prior to the latest community release? If so, you should publicize it, but I think the community response to it will be very negative.

Are you suggesting that the behavior has changed for this from 5.0.37 to 5.0.51 or just guessing? This should be easy for you to check.

Re-opened to check against the latest source.

This is known behavior since 4.0 or earlier. Has been discussed before, suggest checking with Tim to see if he's looked at it.

Thank you for the report.

Verified as described.

To repeat:

1. Patch sql/mysqld.cc:

$diff -u sql/mysqld.cc sql/mysqld.cc2
--- sql/mysqld.cc       2008-01-31 11:11:50.000000000 +0100
+++ sql/mysqld.cc2      2008-01-31 11:11:41.000000000 +0100
@@ -3761,6 +3761,7 @@
       CloseHandle(hEventShutdown);
   }
 #endif
+sleep(20);
   wait_for_signal_thread_to_end();
   clean_up_mutexes();
   my_end(opt_endinfo ? MY_CHECK_ERROR | MY_GIVE_INFO : 0);

2. Start mysqld with help of mysqld_safe
3. issue mysql.server stop
4. Immediately run `kill -9 mysqld_pid` in another shell session
5. See result

Workaround: as root `kill -9 mysqld_safe`

Problem is with mysqld_safe, not with mysql.server script.

One can replace instruction

3. issue mysql.server stop

with

3. issue mysqladmin shutdown

in "How to repeat" section as well

Patch for mysqld_safe to allow it die after repeatable mysqld restarts:

$diff -u ~/bk/mysql-5.0/scripts/mysqld_safe /Users/sveta/Applications/utils/mysql/bin/mysqld_safe_patched.sh
--- /Users/sveta/bk/mysql-5.0/scripts/mysqld_safe       2008-02-01 01:36:54.000000000 +0300
+++ /Users/sveta/Applications/utils/mysql/bin/mysqld_safe_patched.sh    2008-02-01 11:52:34.000000000 +0300
@@ -12,6 +12,12 @@
 
 KILL_MYSQLD=1;
 MYSQLD=
+DIE_ON_REPEATABLE_CRASH=0;
+RETRIES=10;
+TIMEOUT=60;
+RUN_ON_DEATH=
+CRASHES=0;
+RESTART_TIME=`date "+%s"`
 
 trap '' 1 2 3 15                       # we shouldn't let anyone kill us
 
@@ -39,6 +45,10 @@
   --mysqld-version=VERSION   Use "mysqld-VERSION" as mysqld
   --nice=NICE                Set the scheduling priority of mysqld
   --skip-kill-mysqld         Don't try to kill stray mysqld processes
+  --die-on-repeatable-crash  Die in case of repeatable mysqld crashes
+  --retries[=10]             Retries before die
+  --timeout[=60]             Timeout between retries (in seconds)
+  --run-on-death=COMMAND     Run COMMAND on death
 
 All other options are passed to the mysqld program.
 
@@ -94,6 +104,10 @@
       --help)
         usage
         ;;
+         --die-on-repeatable-crash*) DIE_ON_REPEATABLE_CRASH=1; ;;
+         --retries=*) RETRIES=`echo "$arg" | sed -e "s;--retries=;;"` ;;
+         --timeout=*) TIMEOUT=`echo "$arg" | sed -e "s;--timeout=;;"` ;;
+         --run-on-death=*) RUN_ON_DEATH=`echo "$arg" | sed -e "s;--run-on-death=;;"` ;;
       *)
         if test -n "$pick_args"
         then
@@ -106,7 +120,6 @@
   done
 }
 
-
 #
 # First, try to find BASEDIR and ledir (where mysqld is)
 # 
@@ -422,6 +435,20 @@
       I=`expr $I + 1`
     done
   fi
+  if [[ $DIE_ON_REPEATABLE_CRASH ]]
+  then
+         if (( $TIMEOUT < (( - $RESTART_TIME + (RESTART_TIME=`date "+%s"`) )) ))
+         then
+         echo "in";
+               CRASHES=0;
+         fi
+         (( CRASHES++ ))
+         if (( $RETRIES < $CRASHES ))
+         then
+               exec $RUN_ON_DEATH
+               exit 0;
+         fi
+  fi
   echo "`date +'%y%m%d %H:%M:%S'`  mysqld restarted" | tee -a $err_log
 done

patch for mysqld_safe.sh as well

Attachment: mysqld_safe.sh.patch (application/octet-stream, text), 1.67 KiB.

Bug #40423 was marked as duplicate of this one.

Mark,

could you please try patch posted in my comment "[1 Feb 9:56] Sveta Smirnova" and inform us if it solves your problem? Thanks in advance.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

It is on my list of things to do.

It is still on my list of things to do. I don't have time to provide a better response.

We have encountered the same problem when using a redhat clustering solution with mysql.
The cluster tried to shut down MySQL which crashed to perform a failover. But, since mysqld_safe is restarting it - the cluster gets confused and we get a split-brain scenario.

The part of this script that is relevant for us was disabling the auto-restart feature, but the other fixes there seem great as well. 
The script is outdated, and it's time for it to get an overhaul.
Can you please officially release this patch so that it would be a part of an upcoming next version?

This is still a verified bug. Sveta's patch is not applied to any current branch, as far as I can see.

Noted in 5.7.2 changelogs.

If mysqld crashed during a shutdown initiated by /etc/init.d/mysql
stop, mysqld_safe restartd mysqld when it should not have.