Bug #11252 epoch on slave after restore of master dumps does not exist in master
Submitted: 10 Jun 2005 19:02 Modified: 30 Jun 2005 20:22
Reporter: Jonathan Miller Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:5.1.0-wl2325-wl1354-new OS:Linux (Linux)
Assigned to: Tomas Ulin CPU Architecture:Any

[10 Jun 2005 19:02] Jonathan Miller
Description:
Since I had issue setting up a slave from a master dump of a master that was taking transaction, (ref:MySQL Bugs: #11249: Starting slave cluster from masters dump fails with error 4350)
I decided to try a dump from a master that was not taking transactions.

With nothing running on the master cluster I issued a dump command and copied all the dump files to the slave.

On the slave I resrtarted the cluster with --initail, and rebuilt the mysqld var database directory with (rm -rf *) and (mysql_install_db), so that I started with a clean slate.

I started the mysqld process with --skip-slave-start. I created the BANK database and restored all the data files using ndb_restore.

*** Then on the slave in a mysql client I issued:
mysql> SELECT @latest:=MAX(epoch) FROM cluster_replication.apply_status;
+---------------------+
| @latest:=MAX(epoch) |
+---------------------+
|               40836 |
+---------------------+
1 row in set (0.00 sec)

**** On the master I issued:
mysql> SELECT     @file:=SUBSTRING_INDEX(File, '/', -1), @pos:=Position     FROM cluster_replication.binlog_index     WHERE epoch > 40836 ORDER BY epoch ASC LIMIT 1;
Empty set (0.00 sec)

**** and
mysql> SELECT     @file:=SUBSTRING_INDEX(File, '/', -1), @pos:=Position     FROM cluster_replication.binlog_index     WHERE epoch = 40836 ORDER BY epoch ASC LIMIT 1;
Empty set (0.00 sec)

**** Then I issued:
mysql> SELECT epoch,   @file:=SUBSTRING_INDEX(File, '/', -1),@pos:=Position     FROM cluster_replication.binlog_index
| 38142 | master1.000001                        |       86956196 |
| 38143 | master1.000001                        |       86959572 |
+-------+---------------------------------------+----------------+
1268 rows in set (0.01 sec)

epoch 40836 does not exist.

So, I thought that it might be the fact that I was using the -e option for each data restore that I did. 
**** restore command used;
../../../bin/ndb_restore -c ndb10:14000 -n 2 -e -b 3 -m -r ./
../../../bin/ndb_restore -c ndb10:14000 -n 3 -e -b 3  -r ./
../../../bin/ndb_restore -c ndb10:14000 -n 4 -e -b 3  -r ./
../../../bin/ndb_restore -c ndb10:14000 -n 5 -e -b 3  -r ./

**** So then I cleaned up again and tried only using the -e during the metadata restore.

../../../bin/ndb_restore -c ndb10:14000 -n 2 -e -b 3 -m -r ./
../../../bin/ndb_restore -c ndb10:14000 -n 3 -b 3  -r ./
../../../bin/ndb_restore -c ndb10:14000 -n 4 -b 3  -r ./
../../../bin/ndb_restore -c ndb10:14000 -n 5 -b 3  -r ./

**** The the epoch returned was null:

mysql> SELECT @latest:=MAX(epoch) FROM cluster_replication.apply_status;
+---------------------+
| @latest:=MAX(epoch) |
+---------------------+
|                NULL |
+---------------------+

**** To verifiy this I cleaned up and repeated the process.

mysql> SELECT @latest:=MAX(epoch) FROM cluster_replication.apply_status;
+---------------------+
| @latest:=MAX(epoch) |
+---------------------+
|               40836 |
+---------------------+
1 row in set (0.00 sec)

**** Each restore showed no errors:

[ndbdev@ndb10 BACKUP-3]$ ../../../bin/ndb_restore -c ndb10:14000 -n 2 -e -b 3 -m -r ./
Ndb version in backup files: Version 5.1.0
Connected to ndb!!
Successfully restored table BANK/def/ACCOUNT_TYPE
Successfully restored table BANK/def/ACCOUNT_TYPES
Successfully restored table BANK/def/SYSTEM_VALUES
Successfully restored table BANK/def/TRANSACTION
Successfully restored table BANK/def/ACCOUNT
Successfully restored table BANK/def/GL
Successfully restored table test/def/t1
Successfully created index PRIMARY on t1
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT_TYPE(10) fragment 0
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT_TYPES(9) fragment 0
_____________________________________________________
Restoring data in table: BANK/def/SYSTEM_VALUES(8) fragment 0
_____________________________________________________
Restoring data in table: BANK/def/TRANSACTION(7) fragment 0
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT(6) fragment 0
_____________________________________________________
Restoring data in table: BANK/def/GL(5) fragment 0
_____________________________________________________
Restoring data in table: test/def/t1(3) fragment 0
_____________________________________________________
Restoring data in table: cluster_replication/def/apply_status(2) fragment 0
_____________________________________________________
Restoring data in table: sys/def/NDB$EVENTS_0(1) fragment 0
_____________________________________________________
Restoring data in table: sys/def/SYSTAB_0(0) fragment 0
Restored 2019 tuples and 0 log entries

[ndbdev@ndb10 BACKUP-3]$ ../../../bin/ndb_restore -c ndb10:14000 -n 3 -e -b 3  -r ./
Ndb version in backup files: Version 5.1.0
Connected to ndb!!
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT_TYPE(10) fragment 2
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT_TYPES(9) fragment 2
_____________________________________________________
Restoring data in table: BANK/def/SYSTEM_VALUES(8) fragment 2
_____________________________________________________
Restoring data in table: BANK/def/TRANSACTION(7) fragment 2
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT(6) fragment 2
_____________________________________________________
Restoring data in table: BANK/def/GL(5) fragment 2
_____________________________________________________
Restoring data in table: test/def/t1(3) fragment 2
_____________________________________________________
Restoring data in table: cluster_replication/def/apply_status(2) fragment 2
_____________________________________________________
Restoring data in table: sys/def/NDB$EVENTS_0(1) fragment 2
_____________________________________________________
Restoring data in table: sys/def/SYSTAB_0(0) fragment 2
Restored 2140 tuples and 0 log entries

[ndbdev@ndb10 BACKUP-3]$ ../../../bin/ndb_restore -c ndb10:14000 -n 4 -e -b 3  -r ./
Ndb version in backup files: Version 5.1.0
Connected to ndb!!
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT_TYPE(10) fragment 1
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT_TYPES(9) fragment 1
_____________________________________________________
Restoring data in table: BANK/def/SYSTEM_VALUES(8) fragment 1
_____________________________________________________
Restoring data in table: BANK/def/TRANSACTION(7) fragment 1
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT(6) fragment 1
_____________________________________________________
Restoring data in table: BANK/def/GL(5) fragment 1
_____________________________________________________
Restoring data in table: test/def/t1(3) fragment 1
_____________________________________________________
Restoring data in table: cluster_replication/def/apply_status(2) fragment 1
_____________________________________________________
Restoring data in table: sys/def/NDB$EVENTS_0(1) fragment 1
_____________________________________________________
Restoring data in table: sys/def/SYSTAB_0(0) fragment 1
Restored 2119 tuples and 0 log entries

[ndbdev@ndb10 BACKUP-3]$ ../../../bin/ndb_restore -c ndb10:14000 -n 5 -e -b 3  -r ./
Ndb version in backup files: Version 5.1.0
Connected to ndb!!
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT_TYPE(10) fragment 3
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT_TYPES(9) fragment 3
_____________________________________________________
Restoring data in table: BANK/def/SYSTEM_VALUES(8) fragment 3
_____________________________________________________
Restoring data in table: BANK/def/TRANSACTION(7) fragment 3
_____________________________________________________
Restoring data in table: BANK/def/ACCOUNT(6) fragment 3
_____________________________________________________
Restoring data in table: BANK/def/GL(5) fragment 3
_____________________________________________________
Restoring data in table: test/def/t1(3) fragment 3
_____________________________________________________
Restoring data in table: cluster_replication/def/apply_status(2) fragment 3
_____________________________________________________
Restoring data in table: sys/def/NDB$EVENTS_0(1) fragment 3
_____________________________________________________
Restoring data in table: sys/def/SYSTAB_0(0) fragment 3
Restored 2160 tuples and 0 log entries

How to repeat:
follow instruction above.
[13 Jun 2005 8:45] Tomas Ulin
ndb_restore -e will restore the latest epoch included in the backup into apply_status.

binlog_index does only contain entries for epochs that actuall has data.

thus apply_status on the slave may contain an epoch number that is not in the binlog_index on the master

I can't tell from the bug-report if there is anything that is not consistent with this.
[13 Jun 2005 18:03] Tomas Ulin
Thank you for taking the time to write to us, but this is not
a bug. Please double-check the documentation available at
http://www.mysql.com/documentation/ and the instructions on
how to report a bug at http://bugs.mysql.com/how-to-report.php
[30 Jun 2005 20:22] Jonathan Miller
Found that Bug #11674: epoch may not be applied correctly on slave from ndb_restore may have been the cause for this one.