Bug #72475 | Binlog events with binlog_format=MIXED are unconditionally logged in ROW format | ||
---|---|---|---|
Submitted: | 28 Apr 2014 20:28 | Modified: | 5 Dec 2017 14:07 |
Reporter: | Ovais Tariq | Email Updates: | |
Status: | Closed | Impact on me: | |
Category: | MySQL Server: Replication | Severity: | S2 (Serious) |
Version: | 5.5.37, 5.5.38 | OS: | Any |
Assigned to: | CPU Architecture: | Any |
[28 Apr 2014 20:28]
Ovais Tariq
[28 Apr 2014 21:09]
Todd Farmer
Hi Ovais, Thanks for your bug report, but I believe this to be consistent with the documented behavior: " If a statement is logged by row and the session that executed the statement has any temporary tables, logging by row is used for all subsequent statements (except for those accessing temporary tables) until all temporary tables in use by that session are dropped. This is true whether or not any temporary tables are actually logged. Temporary tables cannot be logged using row-based format; thus, once row-based logging is used, all subsequent statements using that table are unsafe. The server approximates this condition by treating all statements executed during the session as unsafe until the session no longer holds any temporary tables. " http://dev.mysql.com/doc/refman/5.6/en/binary-log-mixed.html If you believe this doesn't fully explain the behavior you report, can you help me better understand why? I'll mark this as "Not a Bug" for now, but happy to reconsider with additional information. Thanks!
[28 Apr 2014 22:04]
Ovais Tariq
Todd Farmer, Sadly the explanation does not make complete sense. What does not make sense is why the server approximates all statements as unsafe, if the only statements unsafe are the ones that use the temporary table: "Temporary tables cannot be logged using row-based format; thus, once row-based logging is used, all subsequent statements using that table are unsafe. The server approximates this condition by treating all statements executed during the session as unsafe until the session no longer holds any temporary tables." Since the server already has logic to figure out which statements are true and which are not, then why cannot the same logic be applied to the subsequent statements?
[28 Apr 2014 22:36]
Ovais Tariq
Reopening as a bug there is unintended consequences from the current implementation. So if there are two sessions session-1 and session-2, once session-1 creates a temporary table and uses that, all subsequent statements in session-1 are treated as unsafe and logged in ROW format. However, other session session-2 is an altogether different session and queries are logged based on whether they are safe to use with STATEMENT format or not. However, on downstream slave (with log-slave-updates) the statements from session-2 are also logged in ROW format, every query from every session is logged in ROW format because traditional slave runs everything using a single slave-thread. Corresponding binlog events from the master are: #140429 0:26:47 server id 1 end_log_pos 977 Table_map: `test`.`t2` mapped to number 33 #140429 0:26:47 server id 1 end_log_pos 1053 Update_rows: table id 33 flags: STMT_END_F BINLOG ' p9VeUxMBAAAALAAAANEDAAAAACEAAAAAAAEABHRlc3QAAnQyAAID/gL+IAI= p9VeUxgBAAAATAAAAB0EAAAAACEAAAAAAAEAAv///AQAAAAEZGluZ/wEAAAAB3VwZGF0ZWT8BQAA AARyaW5n/AUAAAAHdXBkYXRlZA== '/*!*/; ### UPDATE `test`.`t2` ### WHERE ### @1=4 /* INT meta=0 nullable=0 is_null=0 */ ### @2='ding' /* STRING(32) meta=65056 nullable=1 is_null=0 */ ### SET ### @1=4 /* INT meta=0 nullable=0 is_null=0 */ ### @2='updated' /* STRING(32) meta=65056 nullable=1 is_null=0 */ ### UPDATE `test`.`t2` ### WHERE ### @1=5 /* INT meta=0 nullable=0 is_null=0 */ ### @2='ring' /* STRING(32) meta=65056 nullable=1 is_null=0 */ ### SET ### @1=5 /* INT meta=0 nullable=0 is_null=0 */ ### @2='updated' /* STRING(32) meta=65056 nullable=1 is_null=0 */ # at 1053 #140429 0:26:47 server id 1 end_log_pos 1080 Xid = 16 COMMIT/*!*/; # at 1080 #140429 0:27:10 server id 1 end_log_pos 1148 Query thread_id=2 exec_time=0 error_code=0 SET TIMESTAMP=1398724030/*!*/; BEGIN /*!*/; # at 1148 # at 1192 #140429 0:27:10 server id 1 end_log_pos 1192 Table_map: `test`.`t2` mapped to number 33 #140429 0:27:10 server id 1 end_log_pos 1240 Write_rows: table id 33 flags: STMT_END_F BINLOG ' vtVeUxMBAAAALAAAAKgEAAAAACEAAAAAAAEABHRlc3QAAnQyAAID/gL+IAI= vtVeUxcBAAAAMAAAANgEAAAAACEAAAAAAAEAAv/8BgAAAA1zaW1wbGVfaW5zZXJ0 '/*!*/; ### INSERT INTO `test`.`t2` ### SET ### @1=6 /* INT meta=0 nullable=0 is_null=0 */ ### @2='simple_insert' /* STRING(32) meta=65056 nullable=1 is_null=0 */ # at 1240 #140429 0:27:10 server id 1 end_log_pos 1267 Xid = 17 COMMIT/*!*/; # at 1267 #140429 0:27:49 server id 1 end_log_pos 1335 Query thread_id=3 exec_time=0 error_code=0 SET TIMESTAMP=1398724069/*!*/; BEGIN /*!*/; # at 1335 #140429 0:27:49 server id 1 end_log_pos 1444 Query thread_id=3 exec_time=0 error_code=0 SET TIMESTAMP=1398724069/*!*/; insert into t2 values(8, 'simple_insert_stmt') /*!*/; As you can see that the last insert has a different thread_id because it is executed in a different session then the one that is working with temp table. Now for the same queries the downstream slave shows the following: #140429 0:26:47 server id 1 end_log_pos 950 Table_map: `test`.`t2` mapped to number 33 #140429 0:26:47 server id 1 end_log_pos 1026 Update_rows: table id 33 flags: STMT_END_F BINLOG ' p9VeUxMBAAAALAAAALYDAAAAACEAAAAAAAEABHRlc3QAAnQyAAID/gL+IAI= p9VeUxgBAAAATAAAAAIEAAAAACEAAAAAAAEAAv///AQAAAAEZGluZ/wEAAAAB3VwZGF0ZWT8BQAA AARyaW5n/AUAAAAHdXBkYXRlZA== '/*!*/; ### UPDATE `test`.`t2` ### WHERE ### @1=4 /* INT meta=0 nullable=0 is_null=0 */ ### @2='ding' /* STRING(32) meta=65056 nullable=1 is_null=0 */ ### SET ### @1=4 /* INT meta=0 nullable=0 is_null=0 */ ### @2='updated' /* STRING(32) meta=65056 nullable=1 is_null=0 */ ### UPDATE `test`.`t2` ### WHERE ### @1=5 /* INT meta=0 nullable=0 is_null=0 */ ### @2='ring' /* STRING(32) meta=65056 nullable=1 is_null=0 */ ### SET ### @1=5 /* INT meta=0 nullable=0 is_null=0 */ ### @2='updated' /* STRING(32) meta=65056 nullable=1 is_null=0 */ # at 1026 #140429 0:26:47 server id 1 end_log_pos 1053 Xid = 10 COMMIT/*!*/; # at 1053 #140429 0:27:10 server id 1 end_log_pos 1112 Query thread_id=2 exec_time=0 error_code=0 SET TIMESTAMP=1398724030/*!*/; BEGIN /*!*/; # at 1112 # at 1156 #140429 0:27:10 server id 1 end_log_pos 1156 Table_map: `test`.`t2` mapped to number 33 #140429 0:27:10 server id 1 end_log_pos 1204 Write_rows: table id 33 flags: STMT_END_F BINLOG ' vtVeUxMBAAAALAAAAIQEAAAAACEAAAAAAAEABHRlc3QAAnQyAAID/gL+IAI= vtVeUxcBAAAAMAAAALQEAAAAACEAAAAAAAEAAv/8BgAAAA1zaW1wbGVfaW5zZXJ0 '/*!*/; ### INSERT INTO `test`.`t2` ### SET ### @1=6 /* INT meta=0 nullable=0 is_null=0 */ ### @2='simple_insert' /* STRING(32) meta=65056 nullable=1 is_null=0 */ # at 1204 #140429 0:27:10 server id 1 end_log_pos 1231 Xid = 14 COMMIT/*!*/; # at 1231 #140429 0:27:49 server id 1 end_log_pos 1290 Query thread_id=3 exec_time=0 error_code=0 SET TIMESTAMP=1398724069/*!*/; BEGIN /*!*/; # at 1290 # at 1334 #140429 0:27:49 server id 1 end_log_pos 1334 Table_map: `test`.`t2` mapped to number 33 #140429 0:27:49 server id 1 end_log_pos 1387 Write_rows: table id 33 flags: STMT_END_F BINLOG ' 5dVeUxMBAAAALAAAADYFAAAAACEAAAAAAAEABHRlc3QAAnQyAAID/gL+IAI= 5dVeUxcBAAAANQAAAGsFAAAAACEAAAAAAAEAAv/8CAAAABJzaW1wbGVfaW5zZXJ0X3N0bXQ= '/*!*/; ### INSERT INTO `test`.`t2` ### SET ### @1=8 /* INT meta=0 nullable=0 is_null=0 */ ### @2='simple_insert_stmt' /* STRING(32) meta=65056 nullable=1 is_null=0 */ # at 1387 #140429 0:27:49 server id 1 end_log_pos 1414 Xid = 16 COMMIT/*!*/; As you can see above the downstream slave wrongly logs the last insert in row format. This is a consequence of the current implementation which you referenced from the manual.
[30 Apr 2014 21:50]
Todd Farmer
Hi Ovais, Thanks for further analysis. In discussion with replication developers, they agree that the current implementation uses a simplification to avoid having to track certain aspects. The impacts of this simplification may have optimization aspects (avoid excessive evaluation or tracking), but also have downstream impacts on the slave. This may also impact in-development features, such as multi-threaded slave, and requires further analysis. In short, this seems like a valid feature request to change the documented behavior.
[1 May 2014 4:07]
Ovais Tariq
Great. That's good to hear. i propose that to deal with temporary tables with MIXED mode replication, the only check should be to see if any query involves reading or writing to a temporary table and then such a query should be logged as ROW. The server is already aware when it executes a statement, as to whether the statement accesses a temporary table, hence it can apply the same logic to the following statements, without having to unconditionally take every statement as an unsafe statement.
[27 May 2014 9:58]
MySQL Verification Team
Hello Ovais, Thank you for the report/feature request!. Verified as described. Thanks, Umesh
[27 Feb 2015 15:32]
Laurynas Biveinis
Bug 72475 patch for 5.7.5 (*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.
Contribution: bug72475-5.7.5.patch (application/octet-stream, text), 187.94 KiB.
[27 Feb 2015 15:33]
Laurynas Biveinis
Patch author George Lorch.
[1 Dec 2016 14:36]
Laurynas Biveinis
Bug 72475 patch for 8.0.0 (*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.
Contribution: bug72475-8.0.0.patch (application/octet-stream, text), 178.04 KiB.
[2 Dec 2016 4:25]
MySQL Verification Team
Thank you Laurynas for the contribution/patch. Thanks, Umesh
[14 Apr 2017 9:11]
Laurynas Biveinis
Bug 72475 fix for 8.0.1 (*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.
Contribution: bug72475-8.0.1.patch (application/octet-stream, text), 171.19 KiB.
[20 Jul 2017 18:50]
Laurynas Biveinis
A fix in 8.0.2 "BUG#24411680 SET BINLOG_FORMAT DOES NOT GENERATE ERROR WHEN THERE ARE TEMPORARY TABLES" pointed out that this fix must also adjust the rules of session binlog format change too when there are open temporary tables. The correct rules now seem to be: forbid change from ROW or MIXED to STATEMENT when there are opened temp tables, as then their CREATE TEMPORARY TABLE might not have been binlogged. All other changes are allowed. A 8.0.2 patch refresh will contain this change. Since the follow-up patch for bug 83003 introduces tracking of binlog format at table create time, with that patch it would be possible to relax this further and only forbid binlog format change if there indeed exists at least one temp table whose CREATE was not binlogged, but currently I'm not planning to implement that, and IMHO moving in and out of SBR with temp tables open seems to be risky with or without the patches anyway.
[4 Aug 2017 19:34]
Laurynas Biveinis
Bug 72475 fix for 8.0.2 (*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.
Contribution: bug72475-8.0.2.patch (application/octet-stream, text), 179.42 KiB.
[4 Aug 2017 19:36]
Laurynas Biveinis
The 8.0.2 implements changed session binlog format change rules as described above. If the change is forbidden, the same error ER_TEMP_TABLE_PREVENTS_SWITCH_OUT_OF_RBR is returned, although now its name and message are misleading.
[5 Dec 2017 14:07]
Margaret Fisher
Posted by developer: Changelog entry added for MySQL 8.0.4: The behavior of mixed format replication has changed with regards to temporary tables. Previously, when mixed binary logging format was in use, if a statement was logged by row and the session that executed the statement had any temporary tables, all subsequent statements were treated as unsafe and logged in row-based format until all temporary tables in use by that session were dropped. Also, on a replication slave with log_slave_updates enabled, row-based logging was incorrectly continued across all subsequent sessions for the duration of the connection, as reported in the bug. Now, when mixed binary logging format is in use, statements that exclusively use temporary tables are not logged. Statements that involve a mix of temporary and non-temporary tables are logged on the master only for the operations on non-temporary tables, and the operations on temporary tables are not logged. The exception is if the creation of a temporary table was recorded in the binary log using statement-based format. In this case, a DROP TEMPORARY TABLE IF EXISTS statement is logged on the master when the temporary table is dropped. With this change in behavior, the remaining statements in the session that do not involve temporary tables no longer need to be treated as unsafe. The safe statements are now logged in statement-based format, and the unsafe statements are logged in row-based format, according to the normal behavior for mixed format replication, regardless of the presence of temporary tables in the session. Also, the issue reported in the bug has been fixed so that subsequent sessions using the connection now use the appropriate logging format for the session, regardless of the format used by earlier sessions. When binlog_format is ROW or STATEMENT, the behavior remains as before. For row-based binary logging format, operations on temporary tables are not logged, with the exception of the DROP TEMPORARY TABLE IF EXISTS statement as for mixed format. For statement-based binary logging format, operations on temporary tables are logged on the master and replicated on the slave, provided that the statements involving temporary tables can be logged safely using statement-based format. binlog_format=STATEMENT is now the only logging mode in which temporary tables are replicated on the slave. You cannot now change the binlog_format setting from ROW or MIXED to STATEMENT at runtime, because any CREATE TEMPORARY TABLE statements will have been omitted from the binary log in the previous mode. You can still switch from STATEMENT to ROW or MIXED format, even when temporary tables have been created. Thanks to George Lorch and Laurynas Biveinis from Percona for the patch. Documentation changes also made in the following topics: https://dev.mysql.com/doc/refman/8.0/en/binary-log-mixed.html https://dev.mysql.com/doc/refman/8.0/en/binary-log-setting.html https://dev.mysql.com/doc/refman/8.0/en/replication-rbr-usage.html https://dev.mysql.com/doc/refman/8.0/en/replication-features-temptables.html