MySQL Bugs: #104765: Binlog compression - handle impossible to replicate transactions

Bug #104765	Binlog compression - handle impossible to replicate transactions
Submitted:	30 Aug 2021 9:41	Modified:	30 Aug 2021 13:47
Reporter:	Sergiu Hlihor	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	8.0.25	OS:	Any
Assigned to:	MySQL Verification Team	CPU Architecture:	Any
Tags:	Binlog Compression, Binlog Replication, Transaction_payload

Description:
Hello,

I'm opening this ticket as a reaction to https://bugs.mysql.com/bug.php?id=103672 which does not state the inconsistency between behaviors. The public documentation states that transactions may be impossible to replicate, but it does not state clearly how dangerous are the side effects. Currently if this happens, it is physically impossible to actually skip the offending transaction from replica because, due to the nature of the implementation, replica cannot actually jump over the binary logs thus breaking completely the replication, with a need to restore from backups or do dirty restores, both time consuming operations. Moreover, since the actual upper limit of of the compressed binary log size is 1GB, the source always know when such a transaction cannot be replicated, yet it still writes it in the logs. Given that in many scenarios, the consistency of replication mechanism is more important than the nature of the transaction, there are 2 possible scenarios that would both be favorable:
  1) Implement flag that would check for compressed transaction size and immediately reject the transaction if compressed size > 1GB with the appropiate error message that would allow a developer to take actions and redesign the transaction strategy. 
  2) Implement mixed binary logs concept where, if compressed size > 1GB, the old uncompressed mechanism is used, that is capable of replicating transactions greater than MAX_ALLOWED_PACKET. 

How to repeat:
No need for description, behavior was already described in https://bugs.mysql.com/bug.php?id=103672

Hi,

> I'm opening this ticket as a reaction to https://bugs.mysql.com/bug.php?id=103672

There is no need to open a new bug as a reaction to another one, you can continue posting to Bug#103672 irrelevant to the status (closed) of that bug. I'm marking this bug as duplicate of Bug#103672 and anything you have to add to it please continue adding directly to Bug#103672.

> The public documentation states that transactions may be impossible to replicate, but it does not state clearly how dangerous are the side effects.

Feel free to reword the "important note" if you think you can write it better, but it clearly states that it might stop replication, not sure how much clearer one can really be.

https://dev.mysql.com/doc/refman/8.0/en/binary-log-transaction-compression.html
"... be aware that a large transaction which can be replicated successfully when binary log transaction compression is not in use, might stop replication due to its size when binary log transaction compression is in use. ..."

Thanks for your efforts and involvement int this bug report.

Please rephrase then from "it will stop" replication to "it will crash replication and rebuilt from backups might be required". Or any other formulation that is truthful about the consequences. "Stop" can easily be interpreted as able to resume without loss of information which is not the case.

The underlying server bug was fixed in 8.0.34 in BUG#33588473. The server now falls back to writing the transaction without compression, if the compressed size would exceed 1 GiB.

It is true that an 8.0.33 server (or older) will produce a corrupted binary log and that replicas may have to recover from backup. It would be better to document this clearly and to advise to not use compression on 8.0.33 and older.

In addition, we the manual should soften the warning for 8.0.34 and up, and state that huge transactions will merely remain non-compressed but not cause replicas to stop.

These documentation changes are now tracked in BUG#38692143.