Bug #29727 Unable to start MySQL after install from RPM
Submitted: 11 Jul 2007 17:24 Modified: 12 Mar 2008 19:42
Reporter: Gerry Reno Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: Installing Severity:S1 (Critical)
Version:5.1.20-beta OS:Linux (Fedora 7)
Assigned to: CPU Architecture:Any

[11 Jul 2007 17:24] Gerry Reno
Description:
On a new Fedora 7 server that has not been running MySQL previously, I have not been able to successfully start MySQL after installing MySQL 5.1.20-beta from RPMS.

Installed all MySQL 5.1.20-beta RPMS.

During the MySQL-server RPM install the server was not able to start.
Subsequently tried reinstalling MySQL-server RPM - same result.

The following symptoms occur:

[root@grp-01-50-51 mysql]# service mysql status
MySQL is not running, but lock exists                      [FAILED]
[root@grp-01-50-51 subsys]# service mysql start
Starting MySQL../etc/init.d/mysql: line 159: kill: (6638) - No such process
                                                           [FAILED]
[root@grp-01-50-51 subsys]# service mysql stop
MySQL manager or server PID file could not be found!       [FAILED]

If you google these errors you will find a lot of forum posts with many suggestions but trying all of these nothing worked to fix the problem.

As root, tried starting /usr/bin/mysqld_safe from command line:

[root@grp-01-50-51 subsys]# mysqld_safe
touch: missing file operand
Try `touch --help' for more information.
chown: missing operand after `mysql'
Try `chown --help' for more information.
070711 13:03:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

Investigating the 'touch' complaint I found only one 'touch' in mysqld_safe:

  touch $err_log; chown $user $err_log

So 'err_log' is not getting defined which is causing all these problems.

How to repeat:
new server
install MySQL 5.1.20-beta from RPMS
[11 Jul 2007 18:48] Gerry Reno
Applied patch from bug 29634 and it stops the 'touch' and 'chown' errors from mysqld_safe but MySQL still will not start.

[root@grp-01-50-51 tmp]# mysqld_safe 
070711 14:37:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

<hit Ctrl-Z here>
[1]+  Stopped                 mysqld_safe
[root@grp-01-50-51 tmp]# 
[root@grp-01-50-51 tmp]# ps -e | grep mysql
 7658 pts/1    00:00:00 mysqld_safe
 7690 pts/1    00:00:00 mysqld
[root@grp-01-50-51 tmp]# service mysql status
MySQL running (7690)                                       [  OK  ]
[root@grp-01-50-51 tmp]# service mysql stop
Shutting down MySQL.                                       [  OK  ]
[root@grp-01-50-51 tmp]# 
[root@grp-01-50-51 tmp]# service mysql start
Starting MySQL.../etc/init.d/mysql: line 159: kill: (7798) - No such process
                                                           [FAILED]
[root@grp-01-50-51 tmp]# 

something is preventing MySQL from starting.

In /etc/init.d/mysql, it is dying in function wait_for_pid():

wait_for_pid () {
  i=0
  while test $i -ne $service_startup_timeout ; do
    sleep 1
    case "$1" in
      'created')
        test -s $pid_file && i='' && break
        kill -0 $2 || break # if the program goes away, stop waiting  <--- DIES HERE
        ;;
      'removed')
        test ! -s $pid_file && i='' && break
        ;;
      *)
        echo "wait_for_pid () usage: wait_for_pid created|removed"
        exit 1
        ;;
    esac
    echo $echo_n ".$echo_c"
    i=`expr $i + 1`
  done

  if test -z "$i" ; then
    log_success_msg
    return 0
  else
    log_failure_msg
    return 1
  fi
}
[11 Jul 2007 19:11] Gerry Reno
Ok I tried starting MySQL using mysqld_safe with the exact same arguments that the init script is using and it starts fine:

[root@grp-01-50-51 tmp]# /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/grp-01-50-51.pid
070711 15:01:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

<check in another terminal and yes mysql is running>

So why is the pid disappearing for the init script?  When it fails I check mysql:
[root@grp-01-50-51 tmp]# ps -e | grep mysql
 7658 pts/1    00:00:00 mysqld_safe
 7690 pts/1    00:00:00 mysqld <defunct>

and yes the pid is dead.  So I can start MySQL successfully from the command line but I cannot start it using the 'service' init script.

So now I'm really confused.  I've backup out the install on the this server and reinstalled everything and still I see these errors.
[11 Jul 2007 19:17] Gerry Reno
I also 'su - mysql' and tried running mysqld_safe with the same arguments as the init script and it too runs fine.
[11 Jul 2007 19:47] Gerry Reno
Put some 'set -x' in the init 'mysql' script:

[root@grp-01-50-51 tmp]# service mysql start
Starting MySQL+ /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/grp-01-50-51.pid
+ ps -e
+ grep mysql
 8807 pts/1    00:00:00 mysql
 8814 pts/1    00:00:00 mysqld_safe
 8816 pts/1    00:00:00 mysql
 8820 pts/1    00:00:00 mysqld_safe
+ wait_for_pid created 8814
+ i=0
+ test 0 -ne 900
+ sleep 1
070711 15:43:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
+ case "$1" in
+ test -s /var/lib/mysql/grp-01-50-51.pid
+ kill -0 8814
+ echo -n .
.++ expr 0 + 1
+ i=1
+ test 1 -ne 900
+ sleep 1
+ case "$1" in
+ test -s /var/lib/mysql/grp-01-50-51.pid
+ kill -0 8814
+ echo -n .
.++ expr 1 + 1
+ i=2
+ test 2 -ne 900
+ sleep 1
070711 15:43:38 mysqld_safe mysqld from pid file /var/lib/mysql/grp-01-50-51.pid ended
+ case "$1" in
+ test -s /var/lib/mysql/grp-01-50-51.pid
+ kill -0 8814
/etc/init.d/mysql: line 159: kill: (8814) - No such process
+ break
+ test -z 2
+ log_failure_msg
+ /etc/redhat-lsb/lsb_log_message failure
                                                           [FAILED]
+ return 1
+ return_value=1
+ set +x

Hope this helps.
[11 Jul 2007 19:52] Gerry Reno
And here is the same output from a working server:

[root@grp-01-50-50 mysql]# service mysql start
Starting MySQL+ /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/grp-01-50-50.pid
+ ps -e
+ grep mysql
 7244 pts/2    00:00:00 mysql
 7251 pts/2    00:00:00 mysqld_safe
 7252 pts/2    00:00:00 mysqld_safe <defunct>
+ wait_for_pid created 7251
+ i=0
+ test 0 -ne 900
+ sleep 1
+ case "$1" in
+ test -s /var/lib/mysql/grp-01-50-50.pid
+ i=
+ break
+ test -z ''
+ log_success_msg
+ /etc/redhat-lsb/lsb_log_message success
                                                           [  OK  ]
+ return 0
+ return_value=0
+ set +x
[12 Jul 2007 1:44] Gerry Reno
Ok, our SELinux policy is set to Enforcing on all our systems.  And I tracked this problem down to a SELinux AVC denial for creation of mysql.sock and the pid file.

The way I fixed the problem was to perform a 'fixfiles relabel' with a reboot.

So I'm not sure where the responsibility lies here, with MySQL or with SELinux so I'm leaving this bug open.  The system originally had the Fedora mysql package installed (but never used) and it was removed using yum.  I think this may somehow have some bearing on the security contexts and maybe a note to that effect should be placed in a faq.

Here is one of the AVC denial alerts:

Summary
    SELinux is preventing /usr/sbin/mysqld (mysqld_t) "create" to mysql.sock
    (var_lib_t).

Detailed Description
    SELinux denied access requested by /usr/sbin/mysqld. It is not expected that
    this access is required by /usr/sbin/mysqld and this access may signal an
    intrusion attempt. It is also possible that the specific version or
    configuration of the application is causing it to require additional access.

Allowing Access
    Sometimes labeling problems can cause SELinux denials.  You could try to
    restore the default system file context for mysql.sock, restorecon -v
    mysql.sock If this does not work, there is currently no automatic way to
    allow this access. Instead,  you can generate a local policy module to allow
    this access - see http://fedora.redhat.com/docs/selinux-faq-fc5/#id2961385
    Or you can disable SELinux protection altogether. Disabling SELinux
    protection is not recommended. Please file a
    http://bugzilla.redhat.com/bugzilla/enter_bug.cgi against this package.

Additional Information        

Source Context                user_u:system_r:mysqld_t
Target Context                user_u:object_r:var_lib_t
Target Objects                mysql.sock [ sock_file ]
Affected RPM Packages         MySQL-server-5.1.20-0.glibc23 [application]
Policy RPM                    selinux-policy-2.6.4-23.fc7
Selinux Enabled               True
Policy Type                   targeted
MLS Enabled                   True
Enforcing Mode                Enforcing
Plugin Name                   plugins.catchall_file
Host Name                     grp-01-50-51
Platform                      Linux grp-01-50-51 2.6.21-1.3228.fc7 #1 SMP Tue
                              Jun 12 15:37:31 EDT 2007 i686 athlon
Alert Count                   37
First Seen                    Wed 11 Jul 2007 12:02:09 PM EDT
Last Seen                     Wed 11 Jul 2007 08:27:32 PM EDT
Local ID                      e25b070f-610b-47f2-a8cb-fc8570c04114
Line Numbers                  

Raw Audit Messages            

avc: denied { create } for comm="mysqld" egid=495 euid=496
exe="/usr/sbin/mysqld" exit=0 fsgid=495 fsuid=496 gid=495 items=0
name="mysql.sock" pid=2686 scontext=user_u:system_r:mysqld_t:s0 sgid=495
subj=user_u:system_r:mysqld_t:s0 suid=496 tclass=sock_file
tcontext=user_u:object_r:var_lib_t:s0 tty=pts1 uid=496
[8 Aug 2007 0:52] Ben Vaughn
I am having similar startup issues to this -- except my SQL will start, just throws the error:

19:50:18 awhq8231 (~) [156] # start_mysql
19:50:21 awhq8231 (~) [157] # usage: touch [-acfhm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]] file ...
usage: chown [-fhv] [-R [-H | -L | -P]] owner[:group] file ...
       chown [-fhv] [-R [-H | -L | -P]] :group file ...
[10 Aug 2007 21:58] Ruben Carrillo
Hi Gerry,

At security level you need to select SELinux and enable the box of Disable SELinux protection for mysqld daemon

Regards

Ruben C.
[17 Aug 2007 20:35] Sarah Novotny
having same problem on centos4 with fresh install of MySQL-server-community-5.0.45-0.rhel4

and, my SELinux policy is disabled.

Sarah
[30 Jan 2008 11:35] Valeriy Kravchuk
Please, try to repeat with newer versions, 5.1.22 and/or 5.0.51a, and SELinux disabled, and inform about the results.
[1 Mar 2008 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[12 Mar 2008 19:42] Susanne Ebrecht
This is a duplicate of bug #29595

All reporters here,

please try to use our packages that you can download from:
http://dev.mysql.com/downloads/

This seems to be a problem from the package maintainer of your OS and is not MySQL related.