Description:
After a system failure or in some cases, if you have ALL conditions belows, mysqld_safe will fail in starting mysqld:
* residual mysqld_safe.pid file
* invoking mysqld_safe without --mysqld and --mysqld-version
* a process has the same pid with the last mysqld_safe's
In these cases, you will get error messages like this, and have mysqld unstarted:
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: write error: Broken pipe
2017-01-14T06:56:36.480646Z mysqld_safe Logging to '/data/mysql_data/data/mysql.err'.
2017-01-14T06:56:36.485739Z mysqld_safe Logging to '/data/mysql_data/data/mysql.err'.
2017-01-14T06:56:36.512756Z mysqld_safe A mysqld process already exists
We found that the grep of line 569 in the mysqld_safe failed, because it using the $MYSQLD wih null value. We invoke mysqld_safe without --mysqld and --mysqld-version, so $MYSQLD has no chance to be assigned an value before the evaluation. The combination results in the final failure.
569 if ps wwwp $PID | grep -v mysqld_safe | grep -- $MYSQLD > /dev/null
Please refer to #How to repeat# for detail( !!!WARNING!!! the operations will kill ALL your mysqld and mysqld_safe ).
How to repeat:
mysql@mysql57:~> grep -n "" ./poc_MYSQLD.sh
1:#!/bin/bash
2:
3:MYSQL_HOME="$1"
4:MY_CNF="$2"
5:
6:MYSQLD_SAFE_PID_FILE=""
7:MYSQLD_SAFE_PID=""
8:TS=""
9:TRY_COUNT=1
10:USER_CONFIRM=""
11:
12:##### function definition #####
13:function fork() { sleep 10 ; }
14:function showMysqlVersion() { $MYSQL_HOME/bin/mysqld --version ; }
15:function startMysql {
16: echo "+ Bingo! we hit $MYSQLD_SAFE_PID after $TRY_COUNT try and `bc<<<$(date +"%s.%N-$TS")` seconds"
17: echo "+ We are going to start mysqld_safe with ./bin/mysqld_safe --defaults-file=$MY_CNF &"
18: echo "+ and you will see the grep failure and pipe broken belows"
19: cd $MYSQL_HOME ; ./bin/mysqld_safe --defaults-file=$MY_CNF &
20: wait
21:}
22:
23:##### main() #####
24:if [ ! -d "$MYSQL_HOME" ] || [ ! -f "$MY_CNF" ] ; then
25: echo "Usage: $0 path_of_mysql_home path_of_my_cnf" && exit 1
26:fi
27:MYSQLD_SAFE_PID_FILE="`$MYSQL_HOME/bin/my_print_defaults --defaults-file=$MY_CNF mysqld | grep -xE -- "--datadir=.*" | cut -d'=' -f2`/mysqld_safe.pid"
28:
29:echo "!!!! WARNING: the script is going to kill all your mysqld_safe and mysqld !!!!!"
30:while [ 1 -eq 1 ] ; do
31: read -p "Do you want to continue (yes/no): " USER_CONFIRM
32: case "$USER_CONFIRM" in
33: "yes") break ;;
34: "no") exit 1 ;;
35: *) continue ;;
36: esac
37:done
38:
39:echo "+ MySQL version: `showMysqlVersion`"
40:
41:if ! test -f "$MYSQLD_SAFE_PID_FILE" ; then
42: echo "+ Cannot get pid from $MYSQLD_SAFE_PID_FILE"
43: exit 1
44:fi
45:MYSQLD_SAFE_PID="`cat $MYSQLD_SAFE_PID_FILE`"
46:echo "+ Get the pid of mysqld_safe: $MYSQLD_SAFE_PID"
47:
48:echo "+ Simulate a crash for mysqld_safe and mysqld"
49:killall -SIGTERM mysqld_safe mysqld
50:
51:echo "+ Start to cracking $MYSQLD_SAFE_PID"
52:TS="`date +'%s.%N'`"
53:if [ "$$" == "$MYSQLD_SAFE_PID" ] || \
54: kill -0 "$MYSQLD_SAFE_PID" &>/dev/null ; then
55: startMysql ; exit 0
56:fi
57:fork &>/dev/null &
58:while [ "$!" != "$MYSQLD_SAFE_PID" ]; do
59: kill -9 "$!"
60: fork &>/dev/null &
61: let TRY_COUNT++
62:done
63:startMysql ; exit 0
mysql@mysql57:~> ./poc_MYSQLD.sh
Usage: ./poc_MYSQLD.sh path_of_mysql_home path_of_my_cnf
mysql@mysql57:~> ./poc_MYSQLD.sh /usr/local/mysql /usr/local/mysql/my.cnf
!!!! WARNING: the script is going to kill all your mysqld_safe and mysqld !!!!!
Do you want to continue (yes/no): yes
+ MySQL version: /usr/local/mysql/bin/mysqld Ver 5.7.17 for linux-glibc2.5 on x86_64 (MySQL Community Server (GPL))
+ Get the pid of mysqld_safe: 12894
+ Simulate a crash for mysqld_safe and mysqld
+ Start to cracking 12894
+ Bingo! we hit 12894 after 1 try and .002467546 seconds
+ We are going to start mysqld_safe with ./bin/mysqld_safe --defaults-file=/usr/local/mysql/my.cnf &
+ and you will see the grep failure and pipe broken belows
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
grep: write error: Broken pipe
2017-01-14T06:56:36.480646Z mysqld_safe Logging to '/data/mysql_data/data/mysql.err'.
2017-01-14T06:56:36.485739Z mysqld_safe Logging to '/data/mysql_data/data/mysql.err'.
2017-01-14T06:56:36.512756Z mysqld_safe A mysqld process already exists
Suggested fix:
We found the intialization codes of $MYSQLD just after the very first evaluation of it. And these lines seems have no dependency to other lines. The suggested fix it to move below code before line 561 to make $MYSQLD initialized before its first evaluation.
725 # If the user doesn't specify a binary, we assume name "mysqld"
726 if test -z "$MYSQLD"
727 then
728 MYSQLD=mysqld
729 fi
Description: After a system failure or in some cases, if you have ALL conditions belows, mysqld_safe will fail in starting mysqld: * residual mysqld_safe.pid file * invoking mysqld_safe without --mysqld and --mysqld-version * a process has the same pid with the last mysqld_safe's In these cases, you will get error messages like this, and have mysqld unstarted: Usage: grep [OPTION]... PATTERN [FILE]... Try `grep --help' for more information. grep: write error: Broken pipe 2017-01-14T06:56:36.480646Z mysqld_safe Logging to '/data/mysql_data/data/mysql.err'. 2017-01-14T06:56:36.485739Z mysqld_safe Logging to '/data/mysql_data/data/mysql.err'. 2017-01-14T06:56:36.512756Z mysqld_safe A mysqld process already exists We found that the grep of line 569 in the mysqld_safe failed, because it using the $MYSQLD wih null value. We invoke mysqld_safe without --mysqld and --mysqld-version, so $MYSQLD has no chance to be assigned an value before the evaluation. The combination results in the final failure. 569 if ps wwwp $PID | grep -v mysqld_safe | grep -- $MYSQLD > /dev/null Please refer to #How to repeat# for detail( !!!WARNING!!! the operations will kill ALL your mysqld and mysqld_safe ). How to repeat: mysql@mysql57:~> grep -n "" ./poc_MYSQLD.sh 1:#!/bin/bash 2: 3:MYSQL_HOME="$1" 4:MY_CNF="$2" 5: 6:MYSQLD_SAFE_PID_FILE="" 7:MYSQLD_SAFE_PID="" 8:TS="" 9:TRY_COUNT=1 10:USER_CONFIRM="" 11: 12:##### function definition ##### 13:function fork() { sleep 10 ; } 14:function showMysqlVersion() { $MYSQL_HOME/bin/mysqld --version ; } 15:function startMysql { 16: echo "+ Bingo! we hit $MYSQLD_SAFE_PID after $TRY_COUNT try and `bc<<<$(date +"%s.%N-$TS")` seconds" 17: echo "+ We are going to start mysqld_safe with ./bin/mysqld_safe --defaults-file=$MY_CNF &" 18: echo "+ and you will see the grep failure and pipe broken belows" 19: cd $MYSQL_HOME ; ./bin/mysqld_safe --defaults-file=$MY_CNF & 20: wait 21:} 22: 23:##### main() ##### 24:if [ ! -d "$MYSQL_HOME" ] || [ ! -f "$MY_CNF" ] ; then 25: echo "Usage: $0 path_of_mysql_home path_of_my_cnf" && exit 1 26:fi 27:MYSQLD_SAFE_PID_FILE="`$MYSQL_HOME/bin/my_print_defaults --defaults-file=$MY_CNF mysqld | grep -xE -- "--datadir=.*" | cut -d'=' -f2`/mysqld_safe.pid" 28: 29:echo "!!!! WARNING: the script is going to kill all your mysqld_safe and mysqld !!!!!" 30:while [ 1 -eq 1 ] ; do 31: read -p "Do you want to continue (yes/no): " USER_CONFIRM 32: case "$USER_CONFIRM" in 33: "yes") break ;; 34: "no") exit 1 ;; 35: *) continue ;; 36: esac 37:done 38: 39:echo "+ MySQL version: `showMysqlVersion`" 40: 41:if ! test -f "$MYSQLD_SAFE_PID_FILE" ; then 42: echo "+ Cannot get pid from $MYSQLD_SAFE_PID_FILE" 43: exit 1 44:fi 45:MYSQLD_SAFE_PID="`cat $MYSQLD_SAFE_PID_FILE`" 46:echo "+ Get the pid of mysqld_safe: $MYSQLD_SAFE_PID" 47: 48:echo "+ Simulate a crash for mysqld_safe and mysqld" 49:killall -SIGTERM mysqld_safe mysqld 50: 51:echo "+ Start to cracking $MYSQLD_SAFE_PID" 52:TS="`date +'%s.%N'`" 53:if [ "$$" == "$MYSQLD_SAFE_PID" ] || \ 54: kill -0 "$MYSQLD_SAFE_PID" &>/dev/null ; then 55: startMysql ; exit 0 56:fi 57:fork &>/dev/null & 58:while [ "$!" != "$MYSQLD_SAFE_PID" ]; do 59: kill -9 "$!" 60: fork &>/dev/null & 61: let TRY_COUNT++ 62:done 63:startMysql ; exit 0 mysql@mysql57:~> ./poc_MYSQLD.sh Usage: ./poc_MYSQLD.sh path_of_mysql_home path_of_my_cnf mysql@mysql57:~> ./poc_MYSQLD.sh /usr/local/mysql /usr/local/mysql/my.cnf !!!! WARNING: the script is going to kill all your mysqld_safe and mysqld !!!!! Do you want to continue (yes/no): yes + MySQL version: /usr/local/mysql/bin/mysqld Ver 5.7.17 for linux-glibc2.5 on x86_64 (MySQL Community Server (GPL)) + Get the pid of mysqld_safe: 12894 + Simulate a crash for mysqld_safe and mysqld + Start to cracking 12894 + Bingo! we hit 12894 after 1 try and .002467546 seconds + We are going to start mysqld_safe with ./bin/mysqld_safe --defaults-file=/usr/local/mysql/my.cnf & + and you will see the grep failure and pipe broken belows Usage: grep [OPTION]... PATTERN [FILE]... Try `grep --help' for more information. grep: write error: Broken pipe 2017-01-14T06:56:36.480646Z mysqld_safe Logging to '/data/mysql_data/data/mysql.err'. 2017-01-14T06:56:36.485739Z mysqld_safe Logging to '/data/mysql_data/data/mysql.err'. 2017-01-14T06:56:36.512756Z mysqld_safe A mysqld process already exists Suggested fix: We found the intialization codes of $MYSQLD just after the very first evaluation of it. And these lines seems have no dependency to other lines. The suggested fix it to move below code before line 561 to make $MYSQLD initialized before its first evaluation. 725 # If the user doesn't specify a binary, we assume name "mysqld" 726 if test -z "$MYSQLD" 727 then 728 MYSQLD=mysqld 729 fi