MySQL Bugs: #102545: Changed binlog default not mentioned in summary

Bug #102545	Changed binlog default not mentioned in summary
Submitted:	9 Feb 2021 14:08	Modified:	19 Feb 2021 15:37
Reporter:	Simon Schneider	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Documentation	Severity:	S2 (Serious)
Version:	8.0	OS:	Red Hat (RHEL 8)
Assigned to:	David Moss	CPU Architecture:	Any

Description:
URL 1: https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html
URL 2: https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_log_bin

The summary of changes doesn't mention that from MySQL 8.0, the binary log is enabled by default. 

The same configuration files in an upgraded installation resulted in the data directory slowly filling up. This led to an unexpected server shutdown. Had the change been mentioned in the upgrading instructions, we could have reacted and the service disruption could have been prevented.

How to repeat:
Find the discrepancy in the documentation:
1) go to URL 1 and search for 'binlog', 'log_bin', 'binary' or related terms. Find nothing.
2) go to URL 2 and find a notice in the second paragraph.

Demonstrate the problem:
1) Set up an instance of MyQSL 5.7
2) Upgrade it to MySQL 8.0
3) Use the server, e.g. by repeatedly dropping and re-creating databases from a backup. This will create files binlog.xxxxxx in the root of the data directory.
4) When the device underlying the data directory is full, the server will no longer operate properly.

Suggested fix:
Repeat the notice from URL 2 in URL 1, and add two recommendations: a) disable the binary log to return to the 5.7 behaviour, or b) keep in mind to regularly purge the binary log.

Hi,

You should always monitor your system, disk space is one of the most important things to monitor. We offer MySQL Enterprise Monitor to monitor your MySQL servers if you are enterprise customer but you can use any of the hundreds available monitoring platforms to monitor your system, each of them support monitoring disk space.

As for documentation, this change is documented:
https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#option_mysqld_...

[quote]
In earlier MySQL versions, binary logging was disabled by default, and was enabled if you specified the --log-bin option. From MySQL 8.0, binary logging is enabled by default, whether or not you specify the --log-bin option. The exception is if you use mysqld to initialize the data directory manually by invoking it with the --initialize or --initialize-insecure option, when binary logging is disabled by default. It is possible to enable binary logging in this case by specifying the --log-bin option. When binary logging is enabled, the log_bin system variable, which shows the status of binary logging on the server, is set to ON. 
[/quote]

Kind regards
Bogdan

Hello, 

my point is: The change is documented, albeit in an obscure location. It is not mentioned in the summary at URL 1, but should be. You need to already know that there is a significant change in order to find the details at URL 2.

As to your recommendations regarding monitoring: That's how we found out that something was wrong in the first place. Our monitoring raised alarms that a server was not responding and that a disk partition was full.

Again, had the change been mentioned in the upgrade instructions, this incident would not have happened. Please treat this as the bug in the upgrade instructions that it is.

Kind regards
Simon Schneider

Hi,

> my point is: The change is documented, albeit in an obscure location.
> It is not mentioned in the summary at URL 1, but should be.

Not all "defaults changes" are mentioned on the upgrade page as "defaults" change through minor versions too, and this is not an incompatible change nor is considered a "significant" change to be mentioned there. That is why it is not documented on that page and why I closed this as "Not a Bug".

> As to your recommendations regarding monitoring: 
> That's how we found out that something was wrong in the first place. 
> Our monitoring raised alarms that a server was not responding and
> that a disk partition was full.

You really need to discuss with your DevOps team but you should get a warning many many days before the server stopped responding, when disk space was getting close to full and not only after the disk was full and server irresponsible. Report that "disk is now full and MySQL is not responding" is kinda useless and defeats the purpose of monitoring :( as when everything stops working you don't need monitoring to tell you it did, you can usually know by your phones getting hot and your inbox getting full.

MySQL is *very* susceptible to "full disk" situation and this is something you must monitor closely so you can intervene *before* disk is full and not after it already happens.

all best
Bogdan

p.s.
I will ask dev team to reconsider adding this info to the upgrade page but that's best I can do

Hello,

thank you for reaching out. I'm aware that my situation is rare: An SQL script not developed in house, but with DDL statements and unexpected behaviour. For diagnosis, we spun up a new virtual server and, to imitate the production server, set up an upgraded MySQL installation. Then we repeatedly restored a database from backup, ran the script, found a new problem, edited the script, rinse and repeat. In a few hours, the binlog had exhausted the disk, too quickly for the monitoring to notice the growth.

Needless to say, that took me by surprise. I'm happy to have found the root cause for my problem and that I now know how to avoid it in the future. If the documentation doesn't get changed (see above: rare edge case), that's also fine. I just wanted to make someone aware.

Thanks and bye,
Simon

Hi Simon,

I do appreciate your report and I'm sure others will too as many users use the bugs system in parallel with documentation. On the other hand, the search engines usually merge the results together anyhow so any info available in the bug system is very useful to everyone hence having this report here, even if it is not in the documentation will reach ppl :)

Kind regards
Bogdan

This seems like a legitimate documentation request; assigning it to someone on our team for resolution.

Thanks for bringing this to our attention.

cheers

jon.

Thank you for your feedback, this has been fixed in the docs here:
https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html