Bug #79442 MySQL 5.7 Server start failing after installation
Submitted: 28 Nov 2015 9:59 Modified: 18 Dec 2015 18:39
Reporter: Pradeep Agrawal Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Packaging Severity:S1 (Critical)
Version:5.7 OS:CentOS (CentOS release 6.6 (Final))
Assigned to: Balasubramanian Kandasamy CPU Architecture:Any

[28 Nov 2015 9:59] Pradeep Agrawal
Description:
After instllation MySQL start is failing, please find the details. 
# wget http://repo.mysql.com//mysql57-community-release-el6-7.noarch.rpm
# yum localinstall mysql57-community-release-el6-7.noarch.rpm
# yum install mysql-community-server
[root@pagrawal-oracle12-mysql57-1 ~]# service mysqld start
Initializing MySQL database:  2015-11-28T08:47:47.620414Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-11-28T08:47:47.623820Z 0 [ERROR] Can't change data directory owner to mysql
2015-11-28T08:47:47.623862Z 0 [ERROR] Aborting

                                                           [FAILED]
[root@pagrawal-oracle12-mysql57-1 ~]# service mysqld start
Initializing MySQL database:  2015-11-28T08:48:08.216504Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-11-28T08:48:08.219238Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2015-11-28T08:48:08.219273Z 0 [ERROR] Aborting

                                                           [FAILED]
[root@pagrawal-oracle12-mysql57-1 ~]# service mysqld start
Initializing MySQL database:  2015-11-28T08:48:13.080433Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-11-28T08:48:13.083365Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2015-11-28T08:48:13.083404Z 0 [ERROR] Aborting

                                                           [FAILED]
[root@pagrawal-oracle12-mysql57-1 ~]# service mysqld start
Initializing MySQL database:  2015-11-28T08:48:16.650635Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-11-28T08:48:16.653302Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2015-11-28T08:48:16.653338Z 0 [ERROR] Aborting

                                                           [FAILED]

How to repeat:
Try above repo in cent os 6.6
[28 Nov 2015 15:26] MySQL Verification Team
If you are doing and fresh install (without older data) just remove the data directory and restart the server. Thanks.
[30 Nov 2015 13:03] Terje Røsten
Hi!

This seems similar to https://bugs.mysql.com/bug.php?id=78591

Do you have SELinux enabled?

You can test by doing:

$ selinuxenabled; echo $?
[30 Nov 2015 19:58] Eric Schoen
I have the same issue with CentOS 6.6 and MySQL 5.7.9 community server installed via yum.  

I have SELinux enabled, and have used chcon, restorecon, and semanage to try to set the security context to mysqld_db_t for the nonstandard data directory.  No joy.

I believe I installed this same version of MysQL installed successfully into a CentOS 6.7 VM using the same yum installation steps.  However, the 6.7 image I started with had an older MySQL (5.1) preinstalled.

If I ensure that the data directory does not exist when starting mysqld for the first time, I think it gets created with the right security context, but is unable to change ownership, even though the directory is owned by mysql.

[root@localhost ~]# service mysqld start
MySQL-Datenbank initialisieren:  2015-11-30T19:53:33.967929Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-11-30T19:53:33.970712Z 0 [ERROR] Can't change data directory owner to mysql
2015-11-30T19:53:33.970956Z 0 [ERROR] Aborting

                                                           [FEHLGESCHLAGEN]

(also... not sure why it's decided to be in a German locale, but this is the least of my problems.)

root@localhost ~]# ls -Z -l /data/XXXXX
total 4
drwxr-x--x. 2 unconfined_u:object_r:mysqld_db_t:s0 mysql mysql 4096 Nov 30 20:53 mysql

The init script has created the certificate files before the server stops:

ls -Z -l /data/XXXXX/mysql/
total 32
-rw-------. 1 unconfined_u:object_r:mysqld_db_t:s0 mysql mysql 1679 Nov 30 20:53 ca-key.pem
-rw-r--r--. 1 unconfined_u:object_r:mysqld_db_t:s0 mysql mysql 1070 Nov 30 20:53 ca.pem
-rw-r--r--. 1 unconfined_u:object_r:mysqld_db_t:s0 mysql mysql 1078 Nov 30 20:53 client-cert.pem
-rw-------. 1 unconfined_u:object_r:mysqld_db_t:s0 mysql mysql 1675 Nov 30 20:53 client-key.pem
-rw-------. 1 unconfined_u:object_r:mysqld_db_t:s0 mysql mysql 1679 Nov 30 20:53 private_key.pem
-rw-r--r--. 1 unconfined_u:object_r:mysqld_db_t:s0 mysql mysql  451 Nov 30 20:53 public_key.pem
-rw-r--r--. 1 unconfined_u:object_r:mysqld_db_t:s0 mysql mysql 1078 Nov 30 20:53 server-cert.pem
-rw-------. 1 unconfined_u:object_r:mysqld_db_t:s0 mysql mysql 1679 Nov 30 20:53 server-key.pem
[4 Dec 2015 16:32] Terje Røsten
Hi guys,

thanks for reporting and feedback.

There were in fact two bugs here, next release will fix the first issue.
We also have work in progress to fix the second issue, then service scripts
should work when SELinux is enabled.

Only workaround at the moment is to disable SELinux.
[5 Dec 2015 12:05] Terje Røsten
One more note: 

It's only for fresh installs this is a issue, systems with existing
data is not affected. 

The problem is in initialization process only.
[9 Dec 2015 14:23] Terje Røsten
Hi again guys,

can you please try the following workaround:

a) update to MySQL 5.7.10, it should be available in repos now, a yum update should fix that.

b) adjust SELinux file context on directory /var/lib/mysql-files by doing:

$ semanage fcontext -a -e /var/lib/mysql /var/lib/mysql-files
$ restorecon -R /var/lib/mysql-files

c) Make sure /var/lib/mysql is empty

d) Try to start server:

$ service mysqld start

Does this work for you?
[9 Dec 2015 14:41] Eric Schoen
I can give it a try when I get home; I'm traveling right now and the disk holding my CentOS VM is not with me.  I've had to step back to CentOS 6.6 (from 6.7) to better match the target environment for our software deployment--would this make a difference?

Eric
[9 Dec 2015 14:44] Terje Røsten
It should work for all CentOS 6.x releases.

Thanks for help so far!
[17 Dec 2015 5:19] Balasubramanian Kandasamy
Posted by developer:
 
Thanks for the bug report. We have pushed the fix to mysql-5.7 and trunk.
 
Due to SELinux policy mysqld can't read init-file located elsewhere than
/var/lib/mysql. --initialize wants clean datadir.
    
Move installing of validate password plugin to after initialize
is done and use /var/lib/mysql as directory for init-file option.
    
Use of other directory than /var/lib/mysql caused SELinux to deny
access to file used as argument to --init-file. This broke initscript for
new installations.
[18 Dec 2015 18:39] Paul Dubois
Noted in 5.7.11 changelog.

On SELinux, mysqld --initialize with an --init-file option could fail
to initialize the data directory.
[5 Jan 2016 1:09] Philip Olson
Worth noting: although fixed in 5.7.11, the 5.7.10 Fedora 23 and EL6 RPM's were also updated with the fix; as a *-5.7.10-<literal>2</literal>-*.rpm release to the Yum repository.