MySQL Bugs: #94183: Malformed file names in index file prevent purging of binary logs

Bug #94183	Malformed file names in index file prevent purging of binary logs
Submitted:	3 Feb 2019 14:33	Modified:	21 Feb 2019 8:41
Reporter:	Jaime Lichauco	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Server: Replication	Severity:	S2 (Serious)
Version:	5.6.34, 5.6.40, 5.7.21	OS:	Linux (Amazon RDS)
Assigned to:	MySQL Verification Team	CPU Architecture:	x86
Tags:	binlog, replication

Description:
We have seen cases where malformed files names in the binary log index file prevent subsequent purging of older binary log files.

In RDS MySQL the log_bin_basename is currently /rdsdbdata/log/binlog/mysql-bin-changelog and the log_bin_index is /rdsdbdata/log/binlog/mysql-bin-changelog.index.

What we have occasionally seen in the mysql-bin-changelog.index file are lines that look like this:

/rdsdbdata/log/binlog/mysql-b/rdsdbdata/log/binlog/mysql-bin-changelog.277758 

It is unknown what led to these malformed file names but there is precedent for writing bad content to the index file as documented in https://bugs.mysql.com/bug.php?id=64579.

These bad file names prevent succesffully running the PURGE BINARY LOGS TO command as binary logs are not purged due to the state of the index file.

So far we have seen this happens on MySQL versions 5.6.34, 5.6.40, and 5.7.21.

How to repeat:
Unknown at this time. Investigating on our side to see if we can easily reproduce this issue.

Suggested fix:
There may be some kind of race condition or similar bug within MySQL when writing to the binary log index file that needs to be fixed.

Example binary log index file with three malformed lines

Attachment: mysql-bin-changelog.index (application/octet-stream, text), 222.27 KiB.

Hi,

I cannot reproduce this, and I never seen this in wild. This type of bugs in my experience usually have to do with hardware issues then with bugs in MySQL but, of course, everything is possible. Anyhow without being able to reproduce it, even running a very high lod setup with multiple rotations I'm not able to reproduce anything similar to this.

You say you can see it on 3 different versions, are these three different systems or single system that you upgraded trough time?

thanks
Bogdan

Hello Bogdan,

Thanks for looking into this despite the lack of reproduction steps or more details. All the occurrences of this issue have been on different instances. This has not been a single instance upgraded over time.

I will see if I can find more details about each instance such as any specific hardware configuration and post if I find anything else.

Regards,
Jaime Lichauco

Thanks, that means my idea that it is HW related don't hold water :( .. it's def. a SW problem but I have no way of reproducing for now. If you find any details that help to reproduce, please share

kind regards
Bogdan

Hi,
Invested few more days into reproducing this one without results. Please update if you manage to get more details about how we could reproduce this

kind regards
Bogdan