| Bug #32407 | Impossible to do point-in-time recovery from older binlog | ||
|---|---|---|---|
| Submitted: | 15 Nov 2007 13:10 | Modified: | 19 Dec 2009 8:42 |
| Reporter: | Sven Sandberg | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Replication | Severity: | S3 (Non-critical) |
| Version: | 5.1 | OS: | Any |
| Assigned to: | Sven Sandberg | CPU Architecture: | Any |
| Tags: | base64, binlog, format_description_event, mysqlbinlog | ||
[15 Nov 2007 13:10]
Sven Sandberg
[15 Nov 2007 14:58]
Sven Sandberg
After discussion with Mats and Lars, we agreed that the problem is that events written in base64 need a format description event first, but this is presently not being printed. We should change mysqlbinlog to have three modes of output w.r.t. base64-output: 1. Never print base64 output. A warning will be printed if the part of the binlog being printed contains a row-based event. 2. Print base64 output for row-based events and for the Format_description_log_event. The Format_description_log_event will be written even if the it appears before the range of the binlog being printed (i.e., even if the user passed --start-position=X and the Format_description_log_event is at position Y<=X). 3. Print base64 output for all events. As in mode 2, the Format_description_log_event will be written even if it is not in the specified range. The default mode should probably be 2 (it is the least mode in which all events can be reconstructed, and also the mode most similar to "mixed" mode).
[28 Nov 2007 18:13]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/38727 ChangeSet@1.2650, 2007-11-28 19:13:44+01:00, sven@riska.(none) +3 -0 BUG#32407: Impossible to do point-in-time recovery from older binlog Problem: it is unsafe to read base64-printed events without first reading the Format_description_log_event (FD). Currently, mysqlbinlog cannot print the FD. I made FD able to print themselves, and made mysqlbinlog tell FD events to print themselves.
[4 Dec 2007 19:44]
Mats Kindahl
Since ``BINLOG`` entries contain real replication events, and real replication events contain a originating server id, the events will be applied with the server id set to the originating server id available in the event, *not* with the server id of the server executing the the ``BINLOG`` statement. As far as possible, we want to retain the event origin information intact, even when restoring from a saved binary log. The event origin is critical to identify the exact change being done in a dynamic replication topology, and although this information currently is somewhat weak as an identifier, we are striving to make it a true "global" change identifier. The consequences of this is that if a event is injected into a circular replication topology where a server with the id of the event is not present, the event will circulate forever. In order to avoid re-applying changes already done, WL#3835 will be implemented, but for the time being, one should be careful with using mysqlbinlog for injecting events into a circular replication topology.
[7 Dec 2007 15:18]
Sven Sandberg
Note to the documentation team: the following has changed: - The first BINLOG statement executed by each client must be a Format_description_log_event, otherwise an error message is produced. It would not be safe to execute BINLOG statements without FD events. - mysqlbinlog now outputs a Format_description_log_event in base64 format by default, for the same reason. The Format_description_log_event is printed even if it is outside the range specified with --start-position=X. To turn off printing the Format_description_log_event, pass the flag --base64-output=never; however, if this flag is used on a binlog containing row events, mysqlbinlog will stop with an error message.
[8 Dec 2007 13:31]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/39601 ChangeSet@1.2651, 2007-12-07 16:38:30+01:00, sven@riska.(none) +16 -0 BUG#32407: Impossible to do point-in-time recovery from older binlog Problem: it is unsafe to read base64-printed events without first reading the Format_description_log_event (FD). Currently, mysqlbinlog cannot print the FD. As a side effect, another bug has also been fixed: When mysqlbinlog --start-position=X was specified, no ROLLBACK was printed. I changed this, so that ROLLBACK is always printed. This patch does several things: - Format_description_log_event (FD) now print themselves in base64 format. - mysqlbinlog is now able to print FD events. It has three modes: --base64-output=auto Print row events in base64 output, and print FD event. The FD event is printed even if it is outside the range specified with --start-position, because it would not be safe to read row events otherwise. This is the default. --base64-output=always Like --base64-output=auto, but also print base64 output for query events. This is like the old --base64-output flag, which is also a shorthand for --base64-output=always --base64-output=never Never print base64 output, generate error if row events occur in binlog. This is useful to suppress the FD event in binlogs known not to contain row events (e.g., because BINLOG statement is unsafe, requires root privileges, is not SQL, etc) - the BINLOG statement now handles FD events correctly, by setting the thread's rli's relay log's description_event_for_exec to the loaded event. In fact, executing a BINLOG statement is almost the same as reading an event from a relay log. Before my patch, the code for this was separated (exec_relay_log_event in slave.cc executes events from the relay log, mysql_client_binlog_statement in sql_binlog.cc executes BINLOG statements). I needed to augment mysql_client_binlog_statement to do parts of what exec_relay_log_event does. Hence, I did a small refactoring and moved parts of exec_relay_log_event to a new function, which I named apply_event_and_update_pos. apply_event_and_update_pos is called both from exec_relay_log_event and from mysql_client_binlog_statement. - When a non-FD event is executed in a BINLOG statement, without previously executing a FD event in a BINLOG statement, it generates an error, because that's unsafe. I took a new error code for that: ER_NO_FORMAT_DESCRIPTION_EVENT_BEFORE_BINLOG_STATEMENTS. In order to get a decent error message containing the name of the event, I added the class method char* Log_event::get_type_str(Log_event_type type), which returns a string name for the given Log_event_type. This is just like the existing char* Log_event::get_type_str(), except it is a class method that takes the log event type as parameter. I also added PRE_GA_*_ROWS_LOG_EVENT to Log_event::get_type_str(), so that names of old rows event are properly printed. - When reading an event, I added a check that the event type is known by the current Format_description_log_event. Without this, it may crash on bad input (and I was struck by this several times). - I patched the following test cases, which all contain BINLOG statements for row events which must be preceded by BINLOG statements for FD events: - rpl_bug31076 While I was here, I fixed some small things in log_event.cc: - replaced hard-coded 4 by EVENT_TYPE_OFFSET in 3 places - replaced return by DBUG_VOID_RETURN in one place - The name of the logfile can be '-' to indicate stdin. Before my patch, the code just checked if the first character is '-'; now it does a full strcmp(). Probably, all arguments that begin with a - are already handled somewhere else as flags, but I still think it is better that the code reflects what it is supposed to do, with as little dependencies as possible on other parts of the code. If we one day implement that all command line arguments after -- are files (as most unix tools do), then we need this. I also fixed the following in slave.cc: - next_event() was declared twice, and queue_event was not static but should be static (not used outside the file).
[10 Dec 2007 9:41]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/39631 ChangeSet@1.2651, 2007-12-10 10:42:44+01:00, sven@riska.(none) +16 -0 BUG#32407: Impossible to do point-in-time recovery from older binlog Problem: it is unsafe to read base64-printed events without first reading the Format_description_log_event (FD). Currently, mysqlbinlog cannot print the FD. As a side effect, another bug has also been fixed: When mysqlbinlog --start-position=X was specified, no ROLLBACK was printed. I changed this, so that ROLLBACK is always printed. This patch does several things: - Format_description_log_event (FD) now print themselves in base64 format. - mysqlbinlog is now able to print FD events. It has three modes: --base64-output=auto Print row events in base64 output, and print FD event. The FD event is printed even if it is outside the range specified with --start-position, because it would not be safe to read row events otherwise. This is the default. --base64-output=always Like --base64-output=auto, but also print base64 output for query events. This is like the old --base64-output flag, which is also a shorthand for --base64-output=always --base64-output=never Never print base64 output, generate error if row events occur in binlog. This is useful to suppress the FD event in binlogs known not to contain row events (e.g., because BINLOG statement is unsafe, requires root privileges, is not SQL, etc) - the BINLOG statement now handles FD events correctly, by setting the thread's rli's relay log's description_event_for_exec to the loaded event. In fact, executing a BINLOG statement is almost the same as reading an event from a relay log. Before my patch, the code for this was separated (exec_relay_log_event in slave.cc executes events from the relay log, mysql_client_binlog_statement in sql_binlog.cc executes BINLOG statements). I needed to augment mysql_client_binlog_statement to do parts of what exec_relay_log_event does. Hence, I did a small refactoring and moved parts of exec_relay_log_event to a new function, which I named apply_event_and_update_pos. apply_event_and_update_pos is called both from exec_relay_log_event and from mysql_client_binlog_statement. - When a non-FD event is executed in a BINLOG statement, without previously executing a FD event in a BINLOG statement, it generates an error, because that's unsafe. I took a new error code for that: ER_NO_FORMAT_DESCRIPTION_EVENT_BEFORE_BINLOG_STATEMENTS. In order to get a decent error message containing the name of the event, I added the class method char* Log_event::get_type_str(Log_event_type type), which returns a string name for the given Log_event_type. This is just like the existing char* Log_event::get_type_str(), except it is a class method that takes the log event type as parameter. I also added PRE_GA_*_ROWS_LOG_EVENT to Log_event::get_type_str(), so that names of old rows event are properly printed. - When reading an event, I added a check that the event type is known by the current Format_description_log_event. Without this, it may crash on bad input (and I was struck by this several times). - I patched the following test cases, which all contain BINLOG statements for row events which must be preceded by BINLOG statements for FD events: - rpl_bug31076 While I was here, I fixed some small things in log_event.cc: - replaced hard-coded 4 by EVENT_TYPE_OFFSET in 3 places - replaced return by DBUG_VOID_RETURN in one place - The name of the logfile can be '-' to indicate stdin. Before my patch, the code just checked if the first character is '-'; now it does a full strcmp(). Probably, all arguments that begin with a - are already handled somewhere else as flags, but I still think it is better that the code reflects what it is supposed to do, with as little dependencies as possible on other parts of the code. If we one day implement that all command line arguments after -- are files (as most unix tools do), then we need this. I also fixed the following in slave.cc: - next_event() was declared twice, and queue_event was not static but should be static (not used outside the file).
[10 Dec 2007 9:48]
Sven Sandberg
Sorry for the multiple commits, I had mail problems. The current one is from 10:41 2007-12-10.
[10 Dec 2007 11:58]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/39642 ChangeSet@1.2651, 2007-12-07 16:38:30+01:00, sven@riska.(none) +16 -0 BUG#32407: Impossible to do point-in-time recovery from older binlog Problem: it is unsafe to read base64-printed events without first reading the Format_description_log_event (FD). Currently, mysqlbinlog cannot print the FD. As a side effect, another bug has also been fixed: When mysqlbinlog --start-position=X was specified, no ROLLBACK was printed. I changed this, so that ROLLBACK is always printed. This patch does several things: - Format_description_log_event (FD) now print themselves in base64 format. - mysqlbinlog is now able to print FD events. It has three modes: --base64-output=auto Print row events in base64 output, and print FD event. The FD event is printed even if it is outside the range specified with --start-position, because it would not be safe to read row events otherwise. This is the default. --base64-output=always Like --base64-output=auto, but also print base64 output for query events. This is like the old --base64-output flag, which is also a shorthand for --base64-output=always --base64-output=never Never print base64 output, generate error if row events occur in binlog. This is useful to suppress the FD event in binlogs known not to contain row events (e.g., because BINLOG statement is unsafe, requires root privileges, is not SQL, etc) - the BINLOG statement now handles FD events correctly, by setting the thread's rli's relay log's description_event_for_exec to the loaded event. In fact, executing a BINLOG statement is almost the same as reading an event from a relay log. Before my patch, the code for this was separated (exec_relay_log_event in slave.cc executes events from the relay log, mysql_client_binlog_statement in sql_binlog.cc executes BINLOG statements). I needed to augment mysql_client_binlog_statement to do parts of what exec_relay_log_event does. Hence, I did a small refactoring and moved parts of exec_relay_log_event to a new function, which I named apply_event_and_update_pos. apply_event_and_update_pos is called both from exec_relay_log_event and from mysql_client_binlog_statement. - When a non-FD event is executed in a BINLOG statement, without previously executing a FD event in a BINLOG statement, it generates an error, because that's unsafe. I took a new error code for that: ER_NO_FORMAT_DESCRIPTION_EVENT_BEFORE_BINLOG_STATEMENTS. In order to get a decent error message containing the name of the event, I added the class method char* Log_event::get_type_str(Log_event_type type), which returns a string name for the given Log_event_type. This is just like the existing char* Log_event::get_type_str(), except it is a class method that takes the log event type as parameter. I also added PRE_GA_*_ROWS_LOG_EVENT to Log_event::get_type_str(), so that names of old rows event are properly printed. - When reading an event, I added a check that the event type is known by the current Format_description_log_event. Without this, it may crash on bad input (and I was struck by this several times). - I patched the following test cases, which all contain BINLOG statements for row events which must be preceded by BINLOG statements for FD events: - rpl_bug31076 While I was here, I fixed some small things in log_event.cc: - replaced hard-coded 4 by EVENT_TYPE_OFFSET in 3 places - replaced return by DBUG_VOID_RETURN in one place - The name of the logfile can be '-' to indicate stdin. Before my patch, the code just checked if the first character is '-'; now it does a full strcmp(). Probably, all arguments that begin with a - are already handled somewhere else as flags, but I still think it is better that the code reflects what it is supposed to do, with as little dependencies as possible on other parts of the code. If we one day implement that all command line arguments after -- are files (as most unix tools do), then we need this. I also fixed the following in slave.cc: - next_event() was declared twice, and queue_event was not static but should be static (not used outside the file).
[13 Dec 2007 17:02]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/39911 ChangeSet@1.2652, 2007-12-13 18:03:56+01:00, sven@riska.(none) +16 -0 BUG#32407: Impossible to do point-in-time recovery from older binlog Problem: it is unsafe to read base64-printed events without first reading the Format_description_log_event (FD). Currently, mysqlbinlog cannot print the FD. As a side effect, another bug has also been fixed: When mysqlbinlog --start-position=X was specified, no ROLLBACK was printed. I changed this, so that ROLLBACK is always printed. This patch does several things: - Format_description_log_event (FD) now print themselves in base64 format. - mysqlbinlog is now able to print FD events. It has three modes: --base64-output=auto Print row events in base64 output, and print FD event. The FD event is printed even if it is outside the range specified with --start-position, because it would not be safe to read row events otherwise. This is the default. --base64-output=always Like --base64-output=auto, but also print base64 output for query events. This is like the old --base64-output flag, which is also a shorthand for --base64-output=always --base64-output=never Never print base64 output, generate error if row events occur in binlog. This is useful to suppress the FD event in binlogs known not to contain row events (e.g., because BINLOG statement is unsafe, requires root privileges, is not SQL, etc) - the BINLOG statement now handles FD events correctly, by setting the thread's rli's relay log's description_event_for_exec to the loaded event. In fact, executing a BINLOG statement is almost the same as reading an event from a relay log. Before my patch, the code for this was separated (exec_relay_log_event in slave.cc executes events from the relay log, mysql_client_binlog_statement in sql_binlog.cc executes BINLOG statements). I needed to augment mysql_client_binlog_statement to do parts of what exec_relay_log_event does. Hence, I did a small refactoring and moved parts of exec_relay_log_event to a new function, which I named apply_event_and_update_pos. apply_event_and_update_pos is called both from exec_relay_log_event and from mysql_client_binlog_statement. - When a non-FD event is executed in a BINLOG statement, without previously executing a FD event in a BINLOG statement, it generates an error, because that's unsafe. I took a new error code for that: ER_NO_FORMAT_DESCRIPTION_EVENT_BEFORE_BINLOG_STATEMENTS. In order to get a decent error message containing the name of the event, I added the class method char* Log_event::get_type_str(Log_event_type type), which returns a string name for the given Log_event_type. This is just like the existing char* Log_event::get_type_str(), except it is a class method that takes the log event type as parameter. I also added PRE_GA_*_ROWS_LOG_EVENT to Log_event::get_type_str(), so that names of old rows event are properly printed. - When reading an event, I added a check that the event type is known by the current Format_description_log_event. Without this, it may crash on bad input (and I was struck by this several times). - I patched the following test cases, which all contain BINLOG statements for row events which must be preceded by BINLOG statements for FD events: - rpl_bug31076 While I was here, I fixed some small things in log_event.cc: - replaced hard-coded 4 by EVENT_TYPE_OFFSET in 3 places - replaced return by DBUG_VOID_RETURN in one place - The name of the logfile can be '-' to indicate stdin. Before my patch, the code just checked if the first character is '-'; now it does a full strcmp(). Probably, all arguments that begin with a - are already handled somewhere else as flags, but I still think it is better that the code reflects what it is supposed to do, with as little dependencies as possible on other parts of the code. If we one day implement that all command line arguments after -- are files (as most unix tools do), then we need this. I also fixed the following in slave.cc: - next_event() was declared twice, and queue_event was not static but should be static (not used outside the file).
[14 Dec 2007 18:00]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/40020 ChangeSet@1.2655, 2007-12-14 19:02:02+01:00, sven@riska.(none) +17 -0 BUG#32407: Impossible to do point-in-time recovery from older binlog Problem: it is unsafe to read base64-printed events without first reading the Format_description_log_event (FD). Currently, mysqlbinlog cannot print the FD. As a side effect, another bug has also been fixed: When mysqlbinlog --start-position=X was specified, no ROLLBACK was printed. I changed this, so that ROLLBACK is always printed. This patch does several things: - Format_description_log_event (FD) now print themselves in base64 format. - mysqlbinlog is now able to print FD events. It has three modes: --base64-output=auto Print row events in base64 output, and print FD event. The FD event is printed even if it is outside the range specified with --start-position, because it would not be safe to read row events otherwise. This is the default. --base64-output=always Like --base64-output=auto, but also print base64 output for query events. This is like the old --base64-output flag, which is also a shorthand for --base64-output=always --base64-output=never Never print base64 output, generate error if row events occur in binlog. This is useful to suppress the FD event in binlogs known not to contain row events (e.g., because BINLOG statement is unsafe, requires root privileges, is not SQL, etc) - the BINLOG statement now handles FD events correctly, by setting the thread's rli's relay log's description_event_for_exec to the loaded event. In fact, executing a BINLOG statement is almost the same as reading an event from a relay log. Before my patch, the code for this was separated (exec_relay_log_event in slave.cc executes events from the relay log, mysql_client_binlog_statement in sql_binlog.cc executes BINLOG statements). I needed to augment mysql_client_binlog_statement to do parts of what exec_relay_log_event does. Hence, I did a small refactoring and moved parts of exec_relay_log_event to a new function, which I named apply_event_and_update_pos. apply_event_and_update_pos is called both from exec_relay_log_event and from mysql_client_binlog_statement. - When a non-FD event is executed in a BINLOG statement, without previously executing a FD event in a BINLOG statement, it generates an error, because that's unsafe. I took a new error code for that: ER_NO_FORMAT_DESCRIPTION_EVENT_BEFORE_BINLOG_STATEMENTS. In order to get a decent error message containing the name of the event, I added the class method char* Log_event::get_type_str(Log_event_type type), which returns a string name for the given Log_event_type. This is just like the existing char* Log_event::get_type_str(), except it is a class method that takes the log event type as parameter. I also added PRE_GA_*_ROWS_LOG_EVENT to Log_event::get_type_str(), so that names of old rows event are properly printed. - When reading an event, I added a check that the event type is known by the current Format_description_log_event. Without this, it may crash on bad input (and I was struck by this several times). - I patched the following test cases, which all contain BINLOG statements for row events which must be preceded by BINLOG statements for FD events: - rpl_bug31076 While I was here, I fixed some small things in log_event.cc: - replaced hard-coded 4 by EVENT_TYPE_OFFSET in 3 places - replaced return by DBUG_VOID_RETURN in one place - The name of the logfile can be '-' to indicate stdin. Before my patch, the code just checked if the first character is '-'; now it does a full strcmp(). Probably, all arguments that begin with a - are already handled somewhere else as flags, but I still think it is better that the code reflects what it is supposed to do, with as little dependencies as possible on other parts of the code. If we one day implement that all command line arguments after -- are files (as most unix tools do), then we need this. I also fixed the following in slave.cc: - next_event() was declared twice, and queue_event was not static but should be static (not used outside the file).
[5 Feb 2008 13:03]
Bugs System
Pushed into 5.1.24-rc
[5 Feb 2008 13:07]
Bugs System
Pushed into 6.0.5-alpha
[6 Feb 2008 22:06]
Jon Stephens
Documented in the 5.1.24 and 6.0.5 changelogs as follows:
The --base64-output option for mysqlbinlog was not honored for
all types of events. This interfered in some cases with
performing point-in-time recovery.
NOTE: The information regarding changes in format description events needs to be included in the Internals documentation, preferably by the implementor.
[6 Mar 2008 13:06]
Jon Stephens
Also documented for 5.1.23-ndb-6.2.14.
[31 Mar 2008 14:37]
Jon Stephens
Also documented for 5.1.23-ndb-6.3.11.
[9 Sep 2009 11:09]
James Day
This fix for this bug introduced bug #46640.
[22 Oct 2009 6:35]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091022063126-l0qzirh9xyhp0bpc) (version source revid:alik@sun.com-20091019135554-s1pvptt6i750lfhv) (merge vers: 6.0.14-alpha) (pib:13)
[22 Oct 2009 7:08]
Bugs System
Pushed into 5.5.0-beta (revid:alik@sun.com-20091022060553-znkmxm0g0gm6ckvw) (version source revid:alik@sun.com-20091019131708-bc6pv55x6287a0wc) (merge vers: 5.5.0-beta) (pib:13)
[22 Oct 2009 16:14]
Jon Stephens
Also documented in the 5.5.0 changelog. (Already documented for 6.0.5.) Closed.
[18 Dec 2009 10:35]
Bugs System
Pushed into 5.1.41-ndb-7.1.0 (revid:jonas@mysql.com-20091218102229-64tk47xonu3dv6r6) (version source revid:jonas@mysql.com-20091218095730-26gwjidfsdw45dto) (merge vers: 5.1.41-ndb-7.1.0) (pib:15)
[18 Dec 2009 10:51]
Bugs System
Pushed into 5.1.41-ndb-6.2.19 (revid:jonas@mysql.com-20091218100224-vtzr0fahhsuhjsmt) (version source revid:jonas@mysql.com-20091217101452-qwzyaig50w74xmye) (merge vers: 5.1.41-ndb-6.2.19) (pib:15)
[18 Dec 2009 11:06]
Bugs System
Pushed into 5.1.41-ndb-6.3.31 (revid:jonas@mysql.com-20091218100616-75d9tek96o6ob6k0) (version source revid:jonas@mysql.com-20091217154335-290no45qdins5bwo) (merge vers: 5.1.41-ndb-6.3.31) (pib:15)
[18 Dec 2009 11:20]
Bugs System
Pushed into 5.1.41-ndb-7.0.11 (revid:jonas@mysql.com-20091218101303-ga32mrnr15jsa606) (version source revid:jonas@mysql.com-20091218064304-ezreonykd9f4kelk) (merge vers: 5.1.41-ndb-7.0.11) (pib:15)
[19 Dec 2009 8:42]
Jon Stephens
No new changelog entries required. Closing.
