Bug #94183 Malformed file names in index file prevent purging of binary logs
Submitted: 3 Feb 14:33 Modified: 21 Feb 8:41
Reporter: Jaime Lichauco Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Replication Severity:S2 (Serious)
Version:5.6.34, 5.6.40, 5.7.21 OS:Linux (Amazon RDS)
Assigned to: Bogdan Kecman CPU Architecture:x86
Tags: binlog, replication

[3 Feb 14:33] Jaime Lichauco
Description:
We have seen cases where malformed files names in the binary log index file prevent subsequent purging of older binary log files.

In RDS MySQL the log_bin_basename is currently /rdsdbdata/log/binlog/mysql-bin-changelog and the log_bin_index is /rdsdbdata/log/binlog/mysql-bin-changelog.index.

What we have occasionally seen in the mysql-bin-changelog.index file are lines that look like this:

/rdsdbdata/log/binlog/mysql-b/rdsdbdata/log/binlog/mysql-bin-changelog.277758 

It is unknown what led to these malformed file names but there is precedent for writing bad content to the index file as documented in https://bugs.mysql.com/bug.php?id=64579.

These bad file names prevent succesffully running the PURGE BINARY LOGS TO command as binary logs are not purged due to the state of the index file.

So far we have seen this happens on MySQL versions 5.6.34, 5.6.40, and 5.7.21.

How to repeat:
Unknown at this time. Investigating on our side to see if we can easily reproduce this issue.

Suggested fix:
There may be some kind of race condition or similar bug within MySQL when writing to the binary log index file that needs to be fixed.
[3 Feb 16:01] Jaime Lichauco
Example binary log index file with three malformed lines

Attachment: mysql-bin-changelog.index (application/octet-stream, text), 222.27 KiB.

[21 Feb 0:50] Bogdan Kecman
Hi,

I cannot reproduce this, and I never seen this in wild. This type of bugs in my experience usually have to do with hardware issues then with bugs in MySQL but, of course, everything is possible. Anyhow without being able to reproduce it, even running a very high lod setup with multiple rotations I'm not able to reproduce anything similar to this.

You say you can see it on 3 different versions, are these three different systems or single system that you upgraded trough time?

thanks
Bogdan
[21 Feb 8:41] Jaime Lichauco
Hello Bogdan,

Thanks for looking into this despite the lack of reproduction steps or more details. All the occurrences of this issue have been on different instances. This has not been a single instance upgraded over time.

I will see if I can find more details about each instance such as any specific hardware configuration and post if I find anything else.

Regards,
Jaime Lichauco
[21 Feb 17:20] Bogdan Kecman
Thanks, that means my idea that it is HW related don't hold water :( .. it's def. a SW problem but I have no way of reproducing for now. If you find any details that help to reproduce, please share

kind regards
Bogdan
[25 Feb 9:12] Bogdan Kecman
Hi,
Invested few more days into reproducing this one without results. Please update if you manage to get more details about how we could reproduce this

kind regards
Bogdan