Bug #72851 | Fix for bug 16345265 in 5.6.11 breaks backward compatibility for InnoDB recovery | ||
---|---|---|---|
Submitted: | 3 Jun 2014 14:01 | Modified: | 4 Jun 2014 13:54 |
Reporter: | Alexey Kopytov | Email Updates: | |
Status: | Verified | Impact on me: | |
Category: | MySQL Server: InnoDB storage engine | Severity: | S3 (Non-critical) |
Version: | 5.6 | OS: | Any |
Assigned to: | CPU Architecture: | Any |
[3 Jun 2014 14:01]
Alexey Kopytov
[4 Jun 2014 8:53]
MySQL Verification Team
Hello Alexey, Thank you for the bug report. Thanks, Umesh
[4 Jun 2014 9:26]
Marko Mäkelä
Alexey, your observation looks correct to me. Unfortunately, I do not think that we can fix this bug. The redo logging for compressed tables was broken before this fix. Because of the breakage, we got a hard-to-reproduce corruption in redo log apply when we were testing the option innodb_log_compressed_pages=OFF. We removed this option from the MySQL 5.6.10 GA release because of the problem. The problem was fixed in MySQL 5.6.11, by changing the way how certain compressed page operations are written to the redo log and how the redo log gets applied. I did not realize that this would affect the redo log compatibility when using innodb_log_compressed_pages=ON. So, unfortunately the fix broke our policy that there should not be incompatible file format changes after the GA release. Side note: While working on the fix, I noticed that there is another problem with the logging of MLOG_ZIP_PAGE_REORGANIZE record. Essentially, this record is assuming that the compression algorithm and parameters remain the same across the recovery. The record is not including the zlib compression level. This was fixed in MySQL 5.6.12 by Oracle Bug#16267120 without introducing a redo log incompatibility.
[4 Jun 2014 9:33]
Marko Mäkelä
One more thing: Upgrading from an earlier server version should only be done after performing a slow shutdown on the old server: SET GLOBAL innodb_fast_shutdown=0; and then initiating a shutdown. Or invoke the old server with --innodb-fast-shutdown=0. After a shutdown, the redo log would be empty and the redo log apply would be skipped. After a slow shutdown, the system is even cleaner: there must be no incomplete transactions, and no pending change buffer merge. You cannot really expect a 5.x server to start up after a crash from a 5.(x-1) server. InnoDB could make the life of backup tools easier by adding some format version identifier to the InnoDB redo log header, to prevent problems like this in the future. It would also prevent the problem of getting a nasty error message when an unknown redo log record type is encountered.
[4 Jun 2014 13:54]
Alexey Kopytov
Marko, Thanks for clarifications. I’m wondering if it is possible to fix the bug by simply ignoring reorg_before_insert on recovery? For 5.6.11+, if the server performs a reorg-before-insert operation during runtime, the insert seems to be only redo-logged if the page is successfully reorganized to accommodate the record. That is, reorg_before_insert is always ‘false’ on recovery for 5.6.11+ data, based on my (limited) code analysis and tests. For earlier server versions, ignoring reorg_before_insert on recovery could at least maintain the status quo, i.e. keep redo-logging as broken as it is currently in those versions without introducing new issues. I’m also not sure if the problem addressed by reorg_before_insert is specific to innodb_log_compressed_pages=OFF. If so, it doesn’t apply to 5.1/5.5? I have implemented this idea in XtraBackup and it passes the XtraBackup suite for 5.1, 5.5 and 5.6 based servers. Which of course does not guarantee correctness, I also understand it is an unlikely and probably undocumented (I didn’t check) scenario for the server when a 5.x recovery is performed on redo logs created with 5.(x-1). But apparently people expect it to work, especially when dealing with filesystem snapshots, see https://bugs.launchpad.net/percona-server/+bug/1295672 for example. But for backup tools, it is obviously a much bigger problem, and I agree having some version identifier in the InnoDB redo log header would be a big help.
[30 Aug 2015 12:07]
Marko Mäkelä
I filed Bug#78275 Implement an InnoDB redo log format version identifier that could allow us to avoid this kind of issues in the future. Of course, also in the future, InnoDB developers would have to remember to adjust the format version identifier whenever the redo log format is changed in some way.