MySQL Bugs: #73550: MySQL Enterprise Backup Random Corruption Issues

Bug #73550	MySQL Enterprise Backup Random Corruption Issues
Submitted:	12 Aug 2014 14:39	Modified:	12 Sep 2014 19:20
Reporter:	Josh Loberant	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Enterprise Backup	Severity:	S2 (Serious)
Version:	3.9, 3.10	OS:	Linux (CentOS 6.3)
Assigned to:		CPU Architecture:	Any
Tags:	checksum, corruption, mismatch, mysqlbackup, validate

Description:
We have two backup servers that execute a full backup once a week, and an incremental every night. Every so often, while joining an incremental to the full backup, it fails with InnoDB: Progress in percent: 0 1 2 3 4 5 6 2014-06-26 14:06:22 7f62860e0700  InnoDB: compressed page checksum mismatch (space 3306 page 61869319): stored: 237160982, crc32: 96943498 innodb: 1250043506, none: 3735928559". Obviously, the numbers change, but the checksum error repeats itself, each time in the same general area.
Upon using the --validate option, I found 3 tables to have corrupted pages, but only in the backup, not the production DB.
This issue is sporadic, and sometimes will run properly. What worries me is the fact that the backup software will backup the database, while silently allowing corrupted pages to be saved, rendering the backup void.
Another issue found was even when the full backup was successful, apparently the incremental was corrupted, but since the --validate option does not work on incrementals, I could not prove it.

How to repeat:
Exectute the backup script, and check the validate log output:
/opt/mysql/meb-3.10/bin/mysqlbackup --defaults-file=/etc/.my.backup.cnf --backup-dir=/var/db/backup/full/ --slave-info backup-and-apply-log > /opt/log/mysql/mysql_backup.log.tmp 2>&1
tail -50 /opt/log/mysql/mysql_backup.log.tmp >> /opt/log/mysql/mysql_backup.log
rm /opt/log/mysql/mysql_backup.log.tmp
rm -rf /var/db/backup/tmp/*
/opt/mysql/meb-3.10/bin/mysqlbackup --backup-dir=/var/db/backup/full validate > /opt/log/mysql/mysql_backup_validate.log 2>&1

Suggested fix:
Integrate a checksum checker similar to the validate switch, in both the full and incremental backups, and once again after a full/incremental join is performed.

Thank  you for the report.

Please send us /etc/.my.backup.cnf file, configuration file for the server and indicate if you are using a network filesystem.

Server config file below. We are using local disk storage

[client]
port			= 3306
socket			= /var/run/mysqld/mysqld.sock

[mysqld_safe]
malloc-lib     = /usr/lib64/libtcmalloc_minimal.so

[mysqld]
port			= 3306
skip-external-locking
max_allowed_packet	= 512M
table_open_cache	= 2048
#skip-slave-start	= 1
#plugin-load=thread_pool=thread_pool.so
## Per INFRA-2475
skip-host-cache
skip-name-resolve
#default_time_zone	= US/Central

ignore_db_dir		= lost+found
ignore_db_dir		= lost@002bfound

transaction_isolation	= REPEATABLE-READ
innodb_strict_mode	= 1
log-error		= /opt/log/mysql/mysqld.err
# This server can't handle this becuase of the disks
general_log		= 1
general_log_file	= /opt/log/mysql/query.log
log_warnings		= 1
datadir			= /var/db/mysql
tmpdir			= /opt/tmp
socket			= /var/run/mysqld/mysqld.sock
max_connections		= 1900
back_log		= 500

server-id		= 3008
report-host		= bdb00.cluster01.okcyok1

# Per Oracle: replication is unreliable with a query
# cache enabled
query_cache_type	= 0
log-bin			= mysql-bin
log_slave_updates
binlog_format		= mixed
relay-log		= relay_log
relay-log-index		= relay_log_index
expire_logs_days	= 7
read_only		= 1

innodb_data_file_path	= ibdata1:10M:autoextend
innodb_file_per_table
innodb_file_format	= Barracuda
innodb_log_group_home_dir = /var/db/mysql/
innodb_buffer_pool_size = 30G
innodb_log_file_size	= 3000M
innodb_max_dirty_pages_pct=1
innodb_flush_method	= O_DIRECT
innodb_io_capacity	= 5000
innodb_log_buffer_size	= 8M
innodb_flush_log_at_trx_commit = 1
sync_binlog		= 1
innodb_lock_wait_timeout = 50
innodb_thread_concurrency= 8
[mysqldump]
quick
max_allowed_packet	= 512M

Thank you for the feedback.

Please also send us content of /etc/.my.backup.cnf file (with password and other confidential information removed) and output of `mount -a`

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".