MySQL Bugs: #42151: Adjust Binlog Filename And Position By Epoch Automatically

Bug #42151	Adjust Binlog Filename And Position By Epoch Automatically
Submitted:	16 Jan 2009 9:06	Modified:	7 May 2009 13:31
Reporter:	Mikiya Okuno	Email Updates:
Status:	Won't fix	Impact on me:	None
Category:	MySQL Cluster: Replication	Severity:	S4 (Feature request)
Version:	any	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
Currently, when configuring a new slave with cluster replication using existing data, we should retrieve a binlog filename and a position by the last epoch in the backup. The procedure is described at page:
http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-replication-backups.html

However, this procedure is complicated and the information required can be calculated by the epoch on the slave. So, if it is possible to setup a new slave using a command like below, it's very useful, isn't it?

mysql> CHANGE MASTER TO ...snip... MASTER_EPOCH=12345678;

Now, cluster replication is complicated and difficult to use. However, once such a feature is implemented, it will be even easier than non-cluster replication.

How to repeat:
n/a

Suggested fix:
Maybe need to change BinlogDump command so that the slave can tell an epoch to the master.

The plan is to address this by making a new Cluster Management system that handles these things for you.  This work is now in progress.

BR,

Tomas

= High Level Design =
Extend COM_BINLOG_DUMP with least change so that the slave can tell the latest epoch to the master.

* A user can specify a new master with the following procedure.

# Retrieve a backup from the master using START BACKUP command from mgm.
# Restore a backup to the slave using ndb_restore with -e option.
# Issue CHANGE MASTER TO command without specifying binlog name and position like below:
# Issue START SLAVE

Command details:

ndb_mgm> START BACKUP

shell> ndb_restore -c mgm_hostname -n nodeid1 -b backupid -m -r /path/to/backup
shell> ndb_restore -c mgm_hostname -n nodeid2 -b backupid -r /path/to/backup
.... snip ....
shell> ndb_restore -c mgm_hostname -n nodeidN -b backupid -e -r /path/to/backup

mysql> RESET SLAVE;
mysql> CHANGE MASTER TO master_host='hostname', master_port=3306,
    -> master_user='repl', paster_password='replpass';
mysql> START SLAVE;

It's easy!

* This will not work in multi-master/master-master setups.
* This will do work well in replication-channel-failover setups.
* User can start a secondary channel using the following commands:

mysql> CHANGE MASTER TO master_host='hostname', master_port=3306,
    -> master_user='repl', paster_password='replpass';
mysql> START SLAVE;

It's easy!

* When a filename or a position is specified, the master and the slave behaves like as before.

= Low level design =

== COM_BINLOG_DUMP format ==

=== normal ===
4 byte: binlog pos
2 byte: flags
4 byte: server-id
----->: binlog file name

=== extended ===
4 byte: binlog pos = 0
2 byte: flags & 2
4 byte: server-id
6 byte: "\0EPCH\0" // Magic number
8 byte: epoch number

Because the binlog filename is replaced with the string which its leading letter is NULL, it looks like "no binlog filename specified" for servers without this extension.

== Packet processing ==

S->M: COM_BINLOG_DUMP with EPOCH
M->S: ROTATE event with a new filename and position
M->S: Ordinal log events (RBR)

== Functions ==

=== request_dump() ===
Read an EPOCH from mysql.ndb_apply_status table if the table is populated properly. read_record() doesn't work properly with other storage engines than MyISAM. So mysql_parse() is used here with the following query:

mysql> SELECT MAX(epoch) INTO @latest FROM mysql.ndb_apply_status;

Then, retrieve the user variable from a THD instance. This query is safe to run because it doesn't return any result set; this is why a user variable is used here.

Then, send the information using the extended COM_BINLOG_DUMP packet like in the section above.

=== mysql_parse() ===
Extract a packet and read the EPOCH, then read the file name and position from  mysql.ndb_binlog_index table and call mysql_binlog_send() function with these stuff. mysql_binlog_send() function calls fake_rotate_event(), which sends a new binlog filename and position to the slave. The slave receives ROTATE event and correct the filename and position accordingly.

if(flag&=2 && binlog_pos == 0 && strncmp(packet + 11, "EPCH", 5))
  retrieve an EPOCH from the packet.
  retrieve a filename and position from ndb_binlog_index table,
    using mysql_parse() with the following query.

SELECT SUBSTRING_INDEX(File, '/', -1), Position
 INTO @file, @pos
 FROM mysql.ndb_binlog_index
 WHERE epoch >= @latest
 ORDER BY epoch ASC LIMIT 1;

  Then, retrieve the user variables.

=== mysql_binlog_send() ===
This function is not changed, but it sends ROTATE event properly according to a new filename and position thru fake_rotate_event() function.

== Why not extend CHANGE MASTER TO? ==
* If I'll do this, I should change a lot. Especially, changing a Master_info class design will have a significant impact.
* User will not want to specify EPOCH manually, because it is stored in the system table.

an experimental patch to implement this feature request. Patch for mysql-5.1.30-ndb-6.3.20

Attachment: rep-ndb-mod.patch (application/octet-stream, text), 11.36 KiB.

Patch for mysql-cluster-6.3.23, fixed bugs.

Attachment: rep-ndb-mod.patch (application/octet-stream, text), 11.38 KiB.

The patch above can work on 6.3.20, 6.3.21 and 6.3.22. Because relevant codes are unchanged.

Use currently as documented

This bug is not scheduled to be fixed at this time.