Description:
When I try to enable log-bin on a server that uses FusionIO, both mysql_install_db and mysqld fail with the following error:
/home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64//bin/mysqld: Error reading file './mysql-binlog.index' (Errcode: 5 - Input/output error)
2017-05-01 19:31:32 9018 [ERROR] MYSQL_BIN_LOG::add_log_to_index failed to copy index file to crash safe index file.
2017-05-01 19:31:32 9018 [ERROR] Could not use (null) for logging (error 0). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
2017-05-01 19:31:32 9018 [ERROR] Aborting
Strace shows the following:
access("./mysql-binlog.index", F_OK) = 0
open("./mysql-binlog.index", O_RDWR|O_CREAT, 0660) = 3
fsync(3) = 0
lseek(3, 0, SEEK_END) = 0
lseek(3, 0, SEEK_CUR) = 0
lseek(3, 0, SEEK_END) = 0
...snip...
open("./mysql-binlog.index_crash_safe", O_RDWR|O_CREAT, 0660) = 14
lseek(14, 0, SEEK_CUR) = 0
lseek(3, 0, SEEK_SET) = 0
read(3, 0x7fff4019a7c0, 8192) = -1 EIO (Input/output error)
I can confirm that MySQL was able to create the index file:
$ ls -l /mnt/dfs/geoff
total 110592
-rw-rw----. 1 fusionio fusionio 12582912 May 1 19:31 ibdata1
-rw-rw----. 1 fusionio fusionio 50331648 May 1 19:31 ib_logfile0
-rw-rw----. 1 fusionio fusionio 50331648 May 1 19:31 ib_logfile1
drwx------. 2 fusionio fusionio 0 May 1 19:31 mysql
-rw-rw----. 1 fusionio fusionio 0 May 1 19:31 mysql-binlog.index
-rw-rw----. 1 fusionio fusionio 0 May 1 19:31 mysql-binlog.index_crash_safe
drwx------. 2 fusionio fusionio 0 May 1 19:31 test
If I use a datadir located on a non-FusionIO device on the same server, then I do not see the same error.
lspci shows the following information about the device:
$ sudo lspci | grep storage
08:00.0 Mass storage controller: SanDisk ioMemory HHHL
And it is using the NVMFS file system:
$ df -hT | grep /mnt/dfs
/dev/fioa nvmfs 1.5T 114G 1.4T 8% /mnt/dfs
fio-status shows the following:
$ sudo fio-status
Found 1 VSL driver package:
4.2.1 build 1137 Driver: loaded
Found 1 ioMemory device in this system
Adapter: ioMono (driver 4.2.1)
ioMemory SX300-1600, Product Number:F13-004-1600-CS-0001, SN:<removed>, FIO SN:<removed>
PCIe Power limit threshold: 24.75W
Connected ioMemory modules:
fct0: 08:00.0, Product Number:F13-004-1600-CS-0001, SN:<redacted>
fct0 Attached
ioMemory Adapter Controller, Product Number:F13-004-1600-CS-0001, SN:<redacted>
Last Power Monitor Incident: 71324 sec
PCI:08:00.0, Slot Number:4
Firmware v8.9.1, rev 20150910 Public
1600.00 GBytes device size
Internal temperature: 38.88 degC, max 58.57 degC
Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00%
Contained Virtual Partitions:
fioa: ID:0, UUID:f3620547-2f65-4ffb-8946-299026bd2dea
fioa State: Online, Type: sparse device, Device: /dev/fioa
ID:0, UUID:f3620547-2f65-4ffb-8946-299026bd2dea
1152921472.00 GBytes device size
I am testing on CentOS 7.2, using the MySQL 5.6 binary tarballs downloaded from the official MySQL web site.
I do not see any errors in the system log, so I don't think there are any problems with the drive itself.
How to repeat:
I am using the following configuration file:
[mysqld]
user=fusionio
datadir=/mnt/dfs/geoff
log_error=/home/fusionio/mysqld.err
log-bin=mysql-binlog
And I am running mysql_install_db in the following way:
sudo /home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64/scripts/mysql_install_db --user=fusionio --defaults-file=/home/fusionio/my2.cnf --basedir=/home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64/ --datadir=/mnt/dfs/geoff
And I am running mysqld in the following way:
sudo /home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64/bin/mysqld --defaults-file=/home/fusionio/my2.cnf --user=fusionio --basedir=/home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64/ --datadir=/mnt/dfs/geoff --log-error=/home/fusionio/mysqld.err &
Both of these fail with the following error when using the FusionIO device:
/home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64//bin/mysqld: Error reading file './mysql-binlog.index' (Errcode: 5 - Input/output error)
2017-05-01 19:31:32 9018 [ERROR] MYSQL_BIN_LOG::add_log_to_index failed to copy index file to crash safe index file.
2017-05-01 19:31:32 9018 [ERROR] Could not use (null) for logging (error 0). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
2017-05-01 19:31:32 9018 [ERROR] Aborting
Suggested fix:
Do not fail like this on FusionIO.
Description: When I try to enable log-bin on a server that uses FusionIO, both mysql_install_db and mysqld fail with the following error: /home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64//bin/mysqld: Error reading file './mysql-binlog.index' (Errcode: 5 - Input/output error) 2017-05-01 19:31:32 9018 [ERROR] MYSQL_BIN_LOG::add_log_to_index failed to copy index file to crash safe index file. 2017-05-01 19:31:32 9018 [ERROR] Could not use (null) for logging (error 0). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it. 2017-05-01 19:31:32 9018 [ERROR] Aborting Strace shows the following: access("./mysql-binlog.index", F_OK) = 0 open("./mysql-binlog.index", O_RDWR|O_CREAT, 0660) = 3 fsync(3) = 0 lseek(3, 0, SEEK_END) = 0 lseek(3, 0, SEEK_CUR) = 0 lseek(3, 0, SEEK_END) = 0 ...snip... open("./mysql-binlog.index_crash_safe", O_RDWR|O_CREAT, 0660) = 14 lseek(14, 0, SEEK_CUR) = 0 lseek(3, 0, SEEK_SET) = 0 read(3, 0x7fff4019a7c0, 8192) = -1 EIO (Input/output error) I can confirm that MySQL was able to create the index file: $ ls -l /mnt/dfs/geoff total 110592 -rw-rw----. 1 fusionio fusionio 12582912 May 1 19:31 ibdata1 -rw-rw----. 1 fusionio fusionio 50331648 May 1 19:31 ib_logfile0 -rw-rw----. 1 fusionio fusionio 50331648 May 1 19:31 ib_logfile1 drwx------. 2 fusionio fusionio 0 May 1 19:31 mysql -rw-rw----. 1 fusionio fusionio 0 May 1 19:31 mysql-binlog.index -rw-rw----. 1 fusionio fusionio 0 May 1 19:31 mysql-binlog.index_crash_safe drwx------. 2 fusionio fusionio 0 May 1 19:31 test If I use a datadir located on a non-FusionIO device on the same server, then I do not see the same error. lspci shows the following information about the device: $ sudo lspci | grep storage 08:00.0 Mass storage controller: SanDisk ioMemory HHHL And it is using the NVMFS file system: $ df -hT | grep /mnt/dfs /dev/fioa nvmfs 1.5T 114G 1.4T 8% /mnt/dfs fio-status shows the following: $ sudo fio-status Found 1 VSL driver package: 4.2.1 build 1137 Driver: loaded Found 1 ioMemory device in this system Adapter: ioMono (driver 4.2.1) ioMemory SX300-1600, Product Number:F13-004-1600-CS-0001, SN:<removed>, FIO SN:<removed> PCIe Power limit threshold: 24.75W Connected ioMemory modules: fct0: 08:00.0, Product Number:F13-004-1600-CS-0001, SN:<redacted> fct0 Attached ioMemory Adapter Controller, Product Number:F13-004-1600-CS-0001, SN:<redacted> Last Power Monitor Incident: 71324 sec PCI:08:00.0, Slot Number:4 Firmware v8.9.1, rev 20150910 Public 1600.00 GBytes device size Internal temperature: 38.88 degC, max 58.57 degC Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00% Contained Virtual Partitions: fioa: ID:0, UUID:f3620547-2f65-4ffb-8946-299026bd2dea fioa State: Online, Type: sparse device, Device: /dev/fioa ID:0, UUID:f3620547-2f65-4ffb-8946-299026bd2dea 1152921472.00 GBytes device size I am testing on CentOS 7.2, using the MySQL 5.6 binary tarballs downloaded from the official MySQL web site. I do not see any errors in the system log, so I don't think there are any problems with the drive itself. How to repeat: I am using the following configuration file: [mysqld] user=fusionio datadir=/mnt/dfs/geoff log_error=/home/fusionio/mysqld.err log-bin=mysql-binlog And I am running mysql_install_db in the following way: sudo /home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64/scripts/mysql_install_db --user=fusionio --defaults-file=/home/fusionio/my2.cnf --basedir=/home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64/ --datadir=/mnt/dfs/geoff And I am running mysqld in the following way: sudo /home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64/bin/mysqld --defaults-file=/home/fusionio/my2.cnf --user=fusionio --basedir=/home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64/ --datadir=/mnt/dfs/geoff --log-error=/home/fusionio/mysqld.err & Both of these fail with the following error when using the FusionIO device: /home/fusionio/mysql-5.6.34-linux-glibc2.5-x86_64//bin/mysqld: Error reading file './mysql-binlog.index' (Errcode: 5 - Input/output error) 2017-05-01 19:31:32 9018 [ERROR] MYSQL_BIN_LOG::add_log_to_index failed to copy index file to crash safe index file. 2017-05-01 19:31:32 9018 [ERROR] Could not use (null) for logging (error 0). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it. 2017-05-01 19:31:32 9018 [ERROR] Aborting Suggested fix: Do not fail like this on FusionIO.