Bug #79250 mysqld_pre_systemd fails when datadir has irrelevant content (e.g. lost+found)

Description:
mysql fails to start due to mysqld_pre_systemd running into an error during initial setup.

lxhost ~ # systemctl start mysqld.service
Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
lxhost ~ # systemctl status mysqld.service
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/x86_64-pc-linux-gnu/lib/systemd/system/mysqld.service; disabled; vendor preset: enabled)
   Active: deactivating (final-sigterm) (Result: exit-code)
  Process: 14081 ExecStart=/usr/x86_64-pc-linux-gnu/bin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=1/FAILURE)
  Process: 14061 ExecStartPre=/usr/x86_64-pc-linux-gnu/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/mysqld.service
           └─14084 /usr/x86_64-pc-linux-gnu/bin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid

Nov 12 14:51:09 lxhost mysqld[14081]: 2015-11-12T13:51:09.099714Z 0 [Note] InnoDB: not started
Nov 12 14:51:09 lxhost mysqld[14081]: 2015-11-12T13:51:09.107477Z 0 [Warning] CA certificate ca.pem is self signed.
Nov 12 14:51:09 lxhost mysqld[14081]: 2015-11-12T13:51:09.107529Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
Nov 12 14:51:09 lxhost mysqld[14081]: 2015-11-12T13:51:09.108381Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
Nov 12 14:51:09 lxhost mysqld[14081]: 2015-11-12T13:51:09.108430Z 0 [Note] IPv6 is available.
Nov 12 14:51:09 lxhost mysqld[14081]: 2015-11-12T13:51:09.108441Z 0 [Note]   - '::' resolves to '::';
Nov 12 14:51:09 lxhost mysqld[14081]: 2015-11-12T13:51:09.108447Z 0 [Note] Server socket created on IP: '::'.
Nov 12 14:51:09 lxhost mysqld[14081]: 2015-11-12T13:51:09.108628Z 0 [Warning] Failed to open optimizer cost constant tables
Nov 12 14:51:09 lxhost mysqld[14081]: 2015-11-12T13:51:09.108745Z 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist
Nov 12 14:51:09 lxhost systemd[1]: mysqld.service: Control process exited, code=exited status=1

When running mysqld_pre_systemd by hand you can see the actual problem:

# /usr/x86_64-pc-linux-gnu/bin/mysqld_pre_systemd
2015-11-12T13:51:31.224066Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-11-12T13:51:31.224162Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2015-11-12T13:51:31.224167Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2015-11-12T13:51:31.227758Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
2015-11-12T13:51:31.227845Z 0 [ERROR] Aborting

The directory is not really empty because datadir is on a separate ext4 partition, so there's naturally a lost+found directory in it.

lxhost ~ # ls -la /var/lib/mysql
total 36
drwxr-xr-x  3 mysql mysql 16384 Nov 12 11:34 .
drwxr-xr-x 15 root  root   4096 Nov 12 14:10 ..
drwx------  2 root  root  16384 Nov 30  2009 lost+found

How to repeat:
1. compile from sources
2. install
3. have datadir mounted on a separate partition / or have a .hidden file in it
4. try to start mysql via the provided systemd service

Suggested fix:
Make mysqld_pre_systemd ignore lost+found directories.

Also make it ignore .hidden files since they cause the same error on running mysqld_pre_systemd. Our package manager for example refuses to install empty directories, so we usually create a .hidden file in such directories. Thankfully mysql 5.7.9 creates it's (default) datadir automatically when started via the systemd service file(s) if it doesn't exist, so we don't have a real problem here. But I'd still suggest mysqld_pre_systemd should not error out when the datadir only contains one (or multiple) .hidden files.

Hi!

Thanks for your report. 

This is due to mysqld --initialize, which insist on empty datadir to run
and is not specific systemd or mysqld_pre_systemd.

I agree the lost+found can be an issue, we might be able to create
a white list or something similar.

Thank you for the report.
I agree with reporter and  Terje that some files should be ignored(lost+found, hidden files etc)

Thanks,
Umesh

Noted in 5.7.11, 5.8.0 changelogs.

Previously, mysqld --initialize required the data directory to not
exist or, if it existed, to be empty. Now an existing data directory
is permitted to be nonempty if every entry either has a name that
begins with a period (.) or is named using an --ignore-db-dir option.