MySQL Bugs: #95071: mysqld should ignore special . prefixed snapshot directories during startup

Bug #95071	mysqld should ignore special . prefixed snapshot directories during startup
Submitted:	20 Apr 2019 17:06	Modified:	12 Aug 2019 16:55
Reporter:	Robb Allan	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S1 (Critical)
Version:	8.0.15	OS:	Any
Assigned to:	Sunny Bains	CPU Architecture:	Any

Description:
mysqld silently fails to start.

log entry:

2019-04-20T04:51:21.765851Z 1 [ERROR] [MY-012097] [InnoDB] Table flags are 0x4800 in the data dictionary but the flags in file mysql.ibd  are 0x800!
2019-04-20T04:51:21.765897Z 1 [ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:2370 thread 34430127616
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
04:51:21 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=1
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 321742 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x804347000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fffdfdfcf40 thread_stack 0x46000
0x2c4066e <_Z19my_print_stacktracePhm+0x2e> at /usr/local/libexec/mysqld
0x202f452 <handle_fatal_signal+0x2c2> at /usr/local/libexec/mysqld
0x803d0d4b6 <_pthread_sigmask+0x536> at /lib/libthr.so.3

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

How to repeat:
re-start, same result.

Thanks for your report!

Is this on a fresh datadir created with MySQL 8.0.15 or is this upgrade from previous release?

An upgrade. Have since uninstalled, reinstalled, and receive same error.

Hi,

Are you saying that you did a clean install of 8.0.16 and that it won't start on FreeBSD as I'm not able to reproduce this.

thanks

Yes. Launch errors:

...
2019-06-10T15:12:30.964752Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 4294967279 = ['.zfs/snapshot/21April2019/undo_001', 'undo_001']

2019-06-10T15:12:30.964783Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 4294967294 = ['.zfs/snapshot/21April2019/mysql.ibd', 'mysql.ibd']

2019-06-10T15:12:30.964960Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Failed, retry may succeed.
2019-06-10T15:12:30.965026Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2019-06-10T15:12:30.965479Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-06-10T15:12:30.974927Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-06-10T15:12:30.975582Z 0 [System] [MY-010910] [Server] /usr/local/libexec/mysqld: Shutdown complete (mysqld 8.0.15)  Source distribution.
2019-06-10T15:12:31.6NZ mysqld_safe mysqld from pid file /var/db/mysql/ifr.helical.com.pid ended

Further:

- FreeBSD pkg install of 8.0.15 binary. Installs without issue.

- run "myself-upgrade", fails, deprecated.

- run /usr/local/bin/mysql -> "ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)"

- run "service mysql-server start" -> "Starting mysql." However, /var/db/mysql/..err -> many Tablespace errors, and then "2019-06-10T15:32:25.226748Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Failed, retry may succeed.
2019-06-10T15:32:25.226847Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2019-06-10T15:32:25.227147Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-06-10T15:32:25.234275Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-06-10T15:32:25.234819Z 0 [System] [MY-010910] [Server] /usr/local/libexec/mysqld: Shutdown complete (mysqld 8.0.16)  Source distribution.
2019-06-10T15:32:25.6NZ mysqld_safe mysqld from pid file /var/db/mysql/ifr.helical.com.pid ended"

Steps to reproduce (on any UNIX platform):

$ cat zfs.sh 
#! /bin/bash

set -x
export datadir=/tmp/zfs
rm -rf $datadir
mysqld --no-defaults -I --datadir=$datadir
mkdir -p $datadir/.zfs/snap
cp -a $datadir/* $datadir/.zfs/snap
mysqld --no-defaults  --datadir=$datadir

$ bash zfs.sh
+ export datadir=/tmp/zfs
+ datadir=/tmp/zfs
+ rm -rf /tmp/zfs
+ mysqld --no-defaults -I --datadir=/tmp/zfs
2019-06-12T12:51:04.996195Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.16) initializing of server in progress as process 26774
2019-06-12T12:51:05.657921Z 4 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: et4HN6Lk5F=(
2019-06-12T12:51:06.122415Z 0 [System] [MY-013170] [Server] /usr/sbin/mysqld (mysqld 8.0.16) initializing of server has completed
+ mkdir -p /tmp/zfs/.zfs/snap
+ cp -a /tmp/zfs/auto.cnf /tmp/zfs/ca-key.pem /tmp/zfs/ca.pem /tmp/zfs/client-cert.pem /tmp/zfs/client-key.pem /tmp/zfs/ib_buffer_pool /tmp/zfs/ibdata1 /tmp/zfs/ib_logfile0 /tmp/zfs/ib_logfile1 /tmp/zfs/#innodb_temp /tmp/zfs/mysql /tmp/zfs/mysql.ibd /tmp/zfs/performance_schema /tmp/zfs/private_key.pem /tmp/zfs/public_key.pem /tmp/zfs/server-cert.pem /tmp/zfs/server-key.pem /tmp/zfs/sys /tmp/zfs/undo_001 /tmp/zfs/undo_002 /tmp/zfs/.zfs/snap
+ mysqld --no-defaults --datadir=/tmp/zfs
2019-06-12T12:51:07.726744Z 0 [Warning] [MY-010139] [Server] Changed limits: max_open_files: 1024 (requested 8161)
2019-06-12T12:51:07.726751Z 0 [Warning] [MY-010142] [Server] Changed limits: table_open_cache: 431 (requested 4000)
2019-06-12T12:51:07.932480Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.16) starting as process 26822
2019-06-12T12:51:07.938638Z 1 [ERROR] [MY-012209] [InnoDB] Multiple files found for the same tablespace ID:
2019-06-12T12:51:07.938653Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 1 = ['.zfs/snap/sys/sys_config.ibd', 'sys/sys_config.ibd']

2019-06-12T12:51:07.938661Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 4294967278 = ['.zfs/snap/undo_002', 'undo_002']

2019-06-12T12:51:07.938667Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 4294967279 = ['.zfs/snap/undo_001', 'undo_001']

2019-06-12T12:51:07.938678Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 4294967294 = ['.zfs/snap/mysql.ibd', 'mysql.ibd']

2019-06-12T12:51:07.938691Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Failed, retry may succeed.
2019-06-12T12:51:07.938703Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2019-06-12T12:51:07.938827Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-06-12T12:51:07.938920Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-06-12T12:51:07.939139Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.16)  MySQL Community Server - GPL.

Some more background:

The ZFS file system (used on e.g. Solaris, FreeBSD and Linux) has a snapshot feature where it creates a .zfs/ directory where snapshot of past file systems
are stored. It easy to for end user to browse file history by just browsing the .zfs directory.

However, MySQL server should *not* consider this special .zfs/ directory as valid subdir to search for tablespace files. 

Please let InnoDB ignore such dirs during startup.

Hi

This isn't just on zfs. SAN implementations frequently store volume snapshots in a hidden folder below the volume. We just experienced the same problem on a NetApp Volume (AFF 8080) where snapshots are kept under a folder  ".snapshot" and MySQL 8.0.16 won't start

See below:

== Scenario 1 - Without Snapshots ==

[root@cpfld01lv] # ls -la /mysql-data/
total 20
drwxr-xr-x  2 mysql mysql 12288 Jun 10 15:14 .
drwxr-xr-x 29 root  root   4096 May 30 11:27 ..
drwxrwxrwx  2 root  root   4096 May 30 16:10 .snapshot
[root@cpfld01lv] #

Using below configuration, MySQL starts normally

[root@cpfld01lv] # cat /etc/mysql/my.cnf
[mysqld]
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 1485
basedir         = /usr
datadir         = /mysql-data
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql-8.0
log_error       = /var/log/mysql/mysqld.log
[root@cpfld01lv] #

[root@cpfld01lv] # systemctl start mysql
[root@cpfld01lv] # mysql -s -e "status"
--------------
mysql  Ver 8.0.16-commercial for Linux on x86_64 (MySQL Enterprise Server - Commercial)

Connection id:          9
Current database:
Current user:           root@localhost
SSL:                    Not in use
Current pager:          stdout
Using outfile:          ''
Using delimiter:        ;
Server version:         8.0.16-commercial MySQL Enterprise Server - Commercial
Protocol version:       10
Connection:             Localhost via UNIX socket
Server characterset:    utf8mb4
Db     characterset:    utf8mb4
Client characterset:    utf8mb4
Conn.  characterset:    utf8mb4
UNIX socket:            /var/run/mysqld/mysqld.sock
Uptime:                 1 min 31 sec

Threads: 2  Questions: 9  Slow queries: 0  Opens: 131  Flush tables: 3  Open tables: 35  Queries per second avg: 0.098 

== Scenario 2 - With a snaphot ==

[root@cpfld01lv] # ls -la /mysql-data/.snapshot
total 28
drwxrwxrwx 3 root  root   4096 Jun 10 16:00 .
drwxr-xr-x 6 mysql mysql 12288 Jun 13 10:39 ..
drwxr-xr-x 6 mysql mysql 12288 Jun 10 15:15 bug_review
[root@cpfld01lv] # ls -la /mysql-data/.snapshot/bug_review/
total 167700
drwxr-xr-x 6 mysql mysql    12288 Jun 10 15:15 .
drwxrwxrwx 3 root  root      4096 Jun 10 16:00 ..
-rw-r----- 1 mysql mysql       56 Jun 10 15:14 auto.cnf
-rw-r----- 1 mysql mysql      402 Jun 10 15:15 binlog.000001
-rw-r----- 1 mysql mysql      155 Jun 10 15:15 binlog.000002
-rw-r----- 1 mysql mysql       32 Jun 10 15:15 binlog.index
-rw------- 1 mysql mysql     1680 Jun 10 15:14 ca-key.pem
-rw-r--r-- 1 mysql mysql     1112 Jun 10 15:14 ca.pem
-rw-r--r-- 1 mysql mysql     1112 Jun 10 15:14 client-cert.pem
-rw------- 1 mysql mysql     1680 Jun 10 15:14 client-key.pem
-rw-r----- 1 mysql mysql     3432 Jun 10 15:15 ib_buffer_pool
-rw-r----- 1 mysql mysql 12582912 Jun 10 15:15 ibdata1
-rw-r----- 1 mysql mysql 50331648 Jun 10 15:15 ib_logfile0
-rw-r----- 1 mysql mysql 50331648 Jun 10 15:14 ib_logfile1
-rw-r----- 1 mysql mysql 12582912 Jun 10 15:15 ibtmp1
drwxr-x--- 2 mysql mysql     4096 Jun 10 15:15 #innodb_temp
drwxr-x--- 2 mysql mysql     4096 Jun 10 15:14 mysql
-rw-r----- 1 mysql mysql 24117248 Jun 10 15:15 mysql.ibd
drwxr-x--- 2 mysql mysql     8192 Jun 10 15:14 performance_schema
-rw------- 1 mysql mysql     1680 Jun 10 15:14 private_key.pem
-rw-r--r-- 1 mysql mysql      452 Jun 10 15:14 public_key.pem
-rw-r--r-- 1 mysql mysql     1112 Jun 10 15:14 server-cert.pem
-rw------- 1 mysql mysql     1676 Jun 10 15:14 server-key.pem
drwxr-x--- 2 mysql mysql     4096 Jun 10 15:14 sys
-rw-r----- 1 mysql mysql 10485760 Jun 10 15:15 undo_001
-rw-r----- 1 mysql mysql 10485760 Jun 10 15:15 undo_002
[root@cpfld01lv]

root@cpfld01lv] # systemctl restart mysql
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
[root@cpfld01lv]

The below is an extract of the log on restart: 

[root@cpfld01lv] # tail -25 /var/log/mysql/mysqld.log
2019-06-10T14:14:59.694377Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) starting as process 3861
2019-06-10T14:15:00.359489Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2019-06-10T14:15:00.371463Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
2019-06-10T14:15:00.399224Z 6 [System] [MY-013172] [Server] Received SHUTDOWN from user boot. Shutting down mysqld (Version: 8.0.16-commercial).
2019-06-10T14:15:02.094419Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.16-commercial)  MySQL Enterprise Server - Commercial.
2019-06-10T14:15:02.450617Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) starting as process 3912
2019-06-10T14:15:03.010413Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2019-06-10T14:15:03.035873Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.16-commercial'  socket: '/var/run/mysqld/mysqld.sock'  port: 1485  MySQL Enterprise Server - Commercial.
2019-06-10T14:15:03.159526Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060
2019-06-13T09:39:22.789645Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.16-commercial)  MySQL Enterprise Server - Commercial.
2019-06-13T09:39:23.200311Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) starting as process 30920
2019-06-13T09:39:23.281286Z 1 [ERROR] [MY-012209] [InnoDB] Multiple files found for the same tablespace ID:
2019-06-13T09:39:23.281315Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 1 = ['.snapshot/bug_review/sys/sys_config.ibd', 'sys/sys_config.ibd']

2019-06-13T09:39:23.281329Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 4294967278 = ['.snapshot/bug_review/undo_002', 'undo_002']

2019-06-13T09:39:23.281339Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 4294967279 = ['.snapshot/bug_review/undo_001', 'undo_001']

2019-06-13T09:39:23.281351Z 1 [ERROR] [MY-012202] [InnoDB] Tablespace ID: 4294967294 = ['.snapshot/bug_review/mysql.ibd', 'mysql.ibd']

2019-06-13T09:39:23.281369Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Failed, retry may succeed.
2019-06-13T09:39:23.281410Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2019-06-13T09:39:23.281624Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-06-13T09:39:23.281883Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-06-13T09:39:23.283197Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.16-commercial)  MySQL Enterprise Server - Commercial.

Thanks
Dayo

Hi Dayo!

I think you are right, all . prefixed dirs should be ignored.

Well… not exactly. "-l" is an accepted but undocumented argument on my system (at least in man).

The presence of a ".*" directory doesn't seem to matter.

Run by hand:

mkdir /tmp/zfs
/usr/local/libexec/mysqld  --no-defaults --log-error=/tmp/zfs/log.err --datadir=$datadir --skip-syslog

==> log.err:
2019-06-12T23:12:54.920261Z 0 [System] [MY-010116] [Server] /usr/local/libexec/mysqld (mysqld 8.0.16) starting as process 80132
2019-06-12T23:12:54.932256Z 1 [ERROR] [MY-011011] [Server] Failed to find valid data directory.
2019-06-12T23:12:54.932486Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-06-12T23:12:54.932652Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-06-12T23:12:54.933028Z 0 [System] [MY-010910] [Server] /usr/local/libexec/mysqld: Shutdown complete (mysqld 8.0.16)  Source distribution.

 mkdir tmp/zfs/.zfs
 mkdir tmp/zfs/.zfs/snap
/usr/local/libexec/mysqld  --no-defaults --log-error=/tmp/zfs/log.err --datadir=$datadir --skip-syslog

==> log.err:
2019-06-12T23:13:57.073303Z 0 [System] [MY-010116] [Server] /usr/local/libexec/mysqld (mysqld 8.0.16) starting as process 80146
2019-06-12T23:13:57.088424Z 1 [ERROR] [MY-011011] [Server] Failed to find valid data directory.
2019-06-12T23:13:57.088781Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-06-12T23:13:57.089047Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-06-12T23:13:57.089617Z 0 [System] [MY-010910] [Server] /usr/local/libexec/mysqld: Shutdown complete (mysqld 8.0.16)  Source distribution.

Further: 

Have deleted all snapshots in the .zfs subdirectory.

Still won’t run:

2019-06-13T02:06:38.6NZ mysqld_safe Logging to '/var/db/mysql/ifr.helical.com.err'.
2019-06-13T02:06:38.6NZ mysqld_safe Starting mysqld daemon with databases from /var/db/mysql
 100
2019-06-13T02:06:39.072266Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2019-06-13T02:06:39.072360Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2019-06-13T02:06:39.072410Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2019-06-13T02:06:39.072462Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2019-06-13T02:06:39.075119Z 0 [System] [MY-010116] [Server] /usr/local/libexec/mysqld (mysqld 8.0.16) starting as process 85528
2019-06-13T02:06:39.080799Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2019-06-13T02:06:39.080849Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8_unicode_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2019-06-13T02:06:45.528251Z 1 [ERROR] [MY-011096] [Server] No data dictionary version number found.
2019-06-13T02:06:45.543232Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-06-13T02:06:45.543672Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-06-13T02:06:46.914359Z 0 [System] [MY-010910] [Server] /usr/local/libexec/mysqld: Shutdown complete (mysqld 8.0.16)  Source distribution.
2019-06-13T02:06:46.6NZ mysqld_safe mysqld from pid file /var/db/mysql/ifr.helical.com.pid ended

-I is just short for initialize: 

As seen from the command:

$ mysqld --help --verbose

  -I, --initialize    Create the default database and exit. Create a super user
                      with a random expired password and store it into the log.

Running mysqld on a empty dir without --initialize option will fail.

What is content of your datadir?

Duh. Misread "-l" as lowercase ell, not as uppercase eye.

With that setting, empty directory initializes, server starts. After shutdown and restart, access to server (with /tmp/zfs datadir) works as expected.

However, trying to launch with original datadir still fails with:
2019-06-13T16:23:35.246693Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2019-06-13T16:23:35.246797Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2019-06-13T16:23:35.249573Z 0 [System] [MY-010116] [Server] /usr/local/libexec/mysqld (mysqld 8.0.16) starting as process 98189
2019-06-13T16:23:35.256652Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2019-06-13T16:23:35.256716Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8_unicode_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2019-06-13T16:23:40.727935Z 1 [ERROR] [MY-011096] [Server] No data dictionary version number found.
2019-06-13T16:23:40.731976Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-06-13T16:23:40.732575Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-06-13T16:23:42.054076Z 0 [System] [MY-010910] [Server] /usr/local/libexec/mysqld: Shutdown complete (mysqld 8.0.16)  Source distribution.

I suspect you still have a partially initialized data directory under /tmp/zfs. Worth emptying out the data directory and re-initializing  again but note that it won't start if you have existing snapshots, hence this bug 

The real issue is the conflicting Tablespace IDs on startup in the data directory when *any" hidden snapshot folder has MySQL recognizable files within it. 

Terje's suggestion to ignore any hidden folders in the datadir  on startup should ideally fix this

1). "Terje's suggestion to ignore any hidden folders in the datadir on startup should ideally fix this": how?

2) fails on an empty directory with -I:

robb@ifr /t/zfs> rm -rf ./*

robb@ifr /t/zfs> ls

robb@ifr /t/zfs> /usr/local/libexec/mysqld -I --datadir=/tmp/zfs

2019-06-17T14:40:56.959545Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2019-06-17T14:40:56.959685Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2019-06-17T14:40:56.960030Z 0 [System] [MY-013169] [Server] /usr/local/libexec/mysqld (mysqld 8.0.16) initializing of server in progress as process 55339
2019-06-17T14:40:56.963233Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2019-06-17T14:40:56.963278Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8_unicode_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2019-06-17T14:40:56.965186Z 0 [Warning] [MY-010122] [Server] One can only use the --user switch if running as root
2019-06-17T14:40:57.058653Z 1 [ERROR] [MY-012271] [InnoDB] The innodb_system data file 'ibdata1' must be writable
2019-06-17T14:40:57.058701Z 1 [ERROR] [MY-012278] [InnoDB] The innodb_system data file 'ibdata1' must be writable
2019-06-17T14:40:57.058936Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2019-06-17T14:40:57.058963Z 0 [ERROR] [MY-013236] [Server] Newly created data directory /tmp/zfs/ is unusable. You can safely remove it.
2019-06-17T14:40:57.059991Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-06-17T14:40:57.060428Z 0 [System] [MY-010910] [Server] /usr/local/libexec/mysqld: Shutdown complete (mysqld 8.0.16)  Source distribution.

All tried on On Ubuntu 16.04 

1) Without any files in the hidden folder

ls  /mysql-data/
# ls -a /mysql-data/
.  ..  .snapshot
# ls -a /mysql-data/.snapshot/
.  ..
# mysqld -I --datadir=/mysql-data
# ls  /mysql-data/
auto.cnf    ca.pem           client-key.pem  ibdata1      ib_logfile1   mysql      performance_schema  public_key.pem   server-key.pem  undo_001
ca-key.pem  client-cert.pem  ib_buffer_pool  ib_logfile0  #innodb_temp  mysql.ibd  private_key.pem     server-cert.pem  sys             undo_002
# tail -50f /var/log/mysql/mysqld.log 
2019-06-20T12:01:37.467229Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) initializing of server in progress as process 9465
2019-06-20T12:01:39.980208Z 5 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: v6q,q7H0B!.t
2019-06-20T12:01:41.048397Z 0 [System] [MY-013170] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) initializing of server has completed
# systemctl start mysql
# tail -f /var/log/mysql/mysqld.log 
2019-06-20T12:01:37.467229Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) initializing of server in progress as process 9465
2019-06-20T12:01:39.980208Z 5 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: v6q,q7H0B!.t
2019-06-20T12:01:41.048397Z 0 [System] [MY-013170] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) initializing of server has completed
2019-06-20T12:02:47.639540Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) starting as process 9555
2019-06-20T12:02:48.280370Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2019-06-20T12:02:48.303276Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.16-commercial'  socket: '/var/run/mysqld/mysqld.sock'  port: 1485  MySQL Enterprise Server - Commercial.
2019-06-20T12:02:48.367265Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 3306
#

2)

dayol@cpfld01lv  > ls -l /tmp/
total 8
drwx------ 2 root root 4096 Jun 20 11:56 vmware-root

dayol@cpfld01lv  > mkdir -p  /tmp/data

dayol@cpfld01lv  > ls -l /tmp/
total 12
drwxr-xr-x 2 dayol infraadmin 4096 Jun 20 15:15 data
drwx------ 2 root  root       4096 Jun 20 11:56 vmware-root
dayol@cpfld01lv  > mysqld -I --datadir=/tmp/data --log_error=/tmp/err.log
dayol@cpfld01lv  > tail -20f /tmp/err.log 
2019-06-20T14:16:27.727186Z 0 [Warning] [MY-010139] [Server] Changed limits: max_open_files: 1024 (requested 8161)
2019-06-20T14:16:27.727195Z 0 [Warning] [MY-010142] [Server] Changed limits: table_open_cache: 431 (requested 4000)
2019-06-20T14:16:27.727489Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) initializing of server in progress as process 25341
2019-06-20T14:16:27.729574Z 0 [Warning] [MY-010122] [Server] One can only use the --user switch if running as root
2019-06-20T14:16:31.107866Z 5 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: 0e/)yh:CGCuZ
2019-06-20T14:16:32.837185Z 0 [System] [MY-013170] [Server] /usr/sbin/mysqld (mysqld 8.0.16-commercial) initializing of server has completed

dayol@cpfld01lv  > ls -l /tmp/data/
total 154688
-rw-r----- 1 dayol infraadmin       56 Jun 20 15:16 auto.cnf
-rw------- 1 dayol infraadmin     1676 Jun 20 15:16 ca-key.pem
-rw-r--r-- 1 dayol infraadmin     1112 Jun 20 15:16 ca.pem
-rw-r--r-- 1 dayol infraadmin     1112 Jun 20 15:16 client-cert.pem
-rw------- 1 dayol infraadmin     1676 Jun 20 15:16 client-key.pem
-rw-r----- 1 dayol infraadmin     5374 Jun 20 15:16 ib_buffer_pool
-rw-r----- 1 dayol infraadmin 12582912 Jun 20 15:16 ibdata1
-rw-r----- 1 dayol infraadmin 50331648 Jun 20 15:16 ib_logfile0
-rw-r----- 1 dayol infraadmin 50331648 Jun 20 15:16 ib_logfile1
drwxr-x--- 2 dayol infraadmin     4096 Jun 20 15:16 #innodb_temp
drwxr-x--- 2 dayol infraadmin     4096 Jun 20 15:16 mysql
-rw-r----- 1 dayol infraadmin 24117248 Jun 20 15:16 mysql.ibd
drwxr-x--- 2 dayol infraadmin     4096 Jun 20 15:16 performance_schema
-rw------- 1 dayol infraadmin     1680 Jun 20 15:16 private_key.pem
-rw-r--r-- 1 dayol infraadmin      452 Jun 20 15:16 public_key.pem
-rw-r--r-- 1 dayol infraadmin     1112 Jun 20 15:16 server-cert.pem
-rw------- 1 dayol infraadmin     1676 Jun 20 15:16 server-key.pem
drwxr-x--- 2 dayol infraadmin     4096 Jun 20 15:16 sys
-rw-r----- 1 dayol infraadmin 10485760 Jun 20 15:16 undo_001
-rw-r----- 1 dayol infraadmin 10485760 Jun 20 15:16 undo_002

OTOH, all done on FreeBSD 12.0:

robb@ifr /tmp> freebsd-version
12.0-RELEASE-p1

robb@ifr /tmp> cd /tmp

robb@ifr /tmp> rm -rf ./zfs

robb@ifr /tmp> rm -rf mysql-log.err 

robb@ifr /tmp> ls -al zfs
ls: zfs: No such file or directory
robb@ifr /tmp> mkdir zfs

robb@ifr /tmp> ls -al zfs
total 33
drwxr-xr-x   2 robb  wheel   2 Jul  6 06:40 .
drwxrwxrwt  21 root  wheel  97 Jul  6 06:40 ..

robb@ifr /tmp> ls -ald zfs
drwxr-xr-x  2 robb  wheel  2 Jul  6 06:40 zfs

robb@ifr /tmp> chmod 777 zfs
robb@ifr /tmp> ls -ald zfs
drwxrwxrwx  2 robb  wheel  2 Jul  6 06:40 zfs

robb@ifr /tmp> /usr/local/libexec/mysqld -I --log-error=/tmp/mysql-log.err --datadir=/tmp/mysql/

robb@ifr /tmp> less mysql-log.err
2019-07-06T10:41:36.610145Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead.
2019-07-06T10:41:36.611394Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2019-07-06T10:41:36.615440Z 0 [System] [MY-013169] [Server] /usr/local/libexec/mysqld (mysqld 8.0.16) initializing of server in progress as process 57555
2019-07-06T10:41:36.629528Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2019-07-06T10:41:36.631754Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8_unicode_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2019-07-06T10:41:36.644233Z 0 [ERROR] [MY-010457] [Server] --initialize specified but the data directory has files in it. Aborting.
2019-07-06T10:41:36.646039Z 0 [ERROR] [MY-013236] [Server] Newly created data directory /tmp/mysql/ is unusable. You can safely remove it.
2019-07-06T10:41:36.646668Z 0 [ERROR] [MY-010119] [Server] Aborting
2019-07-06T10:41:36.656636Z 0 [System] [MY-010910] [Server] /usr/local/libexec/mysqld: Shutdown complete (mysqld 8.0.16)  Source distribution.

Any further suggestions as to what to look for to solve this?

Posted by developer:
 
Fixed as of the upcoming 8.0.18 release, and here's the changelog entry:

InnoDB now ignores hidden directories and files during the tablespace
discovery scan that occurs at startup. Hidden directories and files
include those beginning with "." and hidden and system directories and
files on Windows that are identified by attributes.