MySQL Bugs: #52620: Crash can leave null characters in binary log index file

Bug #52620	Crash can leave null characters in binary log index file
Submitted:	6 Apr 2010 15:06	Modified:	22 Feb 2011 14:57
Reporter:	Harrison Fisk	Email Updates:
Status:	Patch queued	Impact on me:	None
Category:	MySQL Server: Replication	Severity:	S3 (Non-critical)
Version:	5.0.84	OS:	Any
Assigned to:	Assigned Account	CPU Architecture:	Any
Tags:	binary log index, INDEX

Description:
The current procedure for rotating the binary log works as follows:

1. Write terminating event to old binary log
2. fdatasync and close the old binary log
3. Create new log file with open() with O_CREAT flag
4. Write header to the new binary log file
5. Call fdatasync on the new binary log file
6. Write the new filename to the index file
7. Call fdatasync on the index file

If there is a total server crash between steps 6/7, then the binary log index file can end up with incorrect information.

The index file will grow during step 6 and since most filesystems will journal metadata changes, the index file size increase will persist across the crash.

When MySQL restarts, the index file will end up with a lot of null characters at the end (the size of the lost write).  MySQL will continue appending file names to the list, and you will end up with an index file similar to:

/binlogs/binary-logs.002497
/binlogs/binary-logs.002498
/binlogs/binary-logs.002499
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@/binlogs/binary-logs.002500
/binlogs/binary-logs.002501
/binlogs/binary-logs.002502

This can then break things that use the binary log index file.

How to repeat:
Do a complete server crash after the write before the fdatasync during log rotation.  Also the filesystem needs to not have flushed the data.  Not very common, but certainly possible.

Suggested fix:
It could be possible to use atomic rename to take care of this.  So the process could be as follows:

1.  Copy the existing .index file
2.  Append the new file name to the .index file and fdatasync
3.  Atomic rename the new file over the old file
4.  fsync the directory to ensure it is persisted

This is being fixed as part of WL#5493.

Fixed in WL#5493.

Are we not going to get URLs for commits any more?

Hi Mark,

  I am sorry about the wrong bug status - it is now fixed.  About
the commit URL, I believe that WL#5493 commits were posted to
commits@lists.mysql.com.

  Going through the commit lists archive I was able to find the
messages that match the timestamp of the revisions that were
merged into the development tree. Please, find them at:

  The main cset:
  - http://lists.mysql.com/commits/127135

  Plus two follow-up csets (post-push fixes):
  - http://lists.mysql.com/commits/128578
  - http://lists.mysql.com/commits/128928

Regards,
Luís

Please check the main cset:
  http://lists.mysql.com/commits/128244