Bug #89423 Contribution by Facebook: Streaming Binlogs
Submitted: 25 Jan 2018 20:14 Modified: 4 Jul 2018 13:58
Reporter: FBContrib Admin Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Replication Severity:S3 (Non-critical)
Version:8.0.3 OS:Any
Assigned to: CPU Architecture:Any

[25 Jan 2018 20:14] FBContrib Admin
Description:
Background innformation provided by Facebook:
Abstract:

Currently 5.6 doesn't suppport compressed binlogs as input to mysqlbinlog.

The original bug link is http://bugs.mysql.com/bug.php?id=49336

Repo: https://github.com/mysql/mysql-server
Patch on top of 8.0.3: https://github.com/mysql/mysql-server/commit/020025d9f277fd3283b8c75024f590857f58fbbd
Facebook commits: https://github.com/facebook/mysql-5.6/commit/e60bd67

This fix allows mysqlbinlog to take multiple compressed files as direct input 
to mysqlbinlog rather than first decompressing the input files and passing 
them to mysqlbinlog.

How to repeat:
See description

Suggested fix:
See contribution code attached
[25 Jan 2018 20:14] FBContrib Admin
Streaming Binlogs 
(*) This code is contributed under the Facebook agreement

Contribution: fb_patch_24.txt (text/plain), 52.06 KiB.

[26 Jun 2018 12:08] Margaret Fisher
Posted by developer:
 
Changelog entry added for MySQL 8.0.12:

  You can now use a shell pipe to supply an archive of compressed binary log files to mysqlbinlog as streamed input.  Previously, the files had to be extracted separately before passing them to mysqlbinlog. Thanks to the Facebook team for this contribution.

In this example, binlog-files_1.gz contains multiple binary log files for processing. The pipeline extracts the contents of binlog-files_1.gz, pipes the binary log files to mysqlbinlog as standard input, and pipes the output of mysqlbinlog into the mysql client for execution:
gzip -cd binlog-files_1.gz | ./mysqlbinlog - | ./mysql -uroot  -p
[4 Jul 2018 13:58] Margaret Fisher
Posted by developer:
 
Documentation added to https://dev.mysql.com/doc/refman/8.0/en/mysqlbinlog.html :

You can also supply multiple binary log files to mysqlbinlog as streamed input using a shell pipe. An archive of compressed binary log files can be decompressed and provided directly to mysqlbinlog. In this example, binlog-files_1.gz contains multiple binary log files for processing. The pipeline extracts the contents of binlog-files_1.gz, pipes the binary log files to mysqlbinlog as standard input, and pipes the output of mysqlbinlog into the mysql client for execution:
gzip -cd binlog-files_1.gz | ./mysqlbinlog - | ./mysql -uroot -p

You can specify more than one archive file, for example:
gzip -cd binlog-files_1.gz binlog-files_2.gz | ./mysqlbinlog - | ./mysql -uroot -p 

For streamed input, do not use --stop-position, because mysqlbinlog cannot identify the last log file to apply this option.