MySQL Bugs: #26913: Three written LCPs during node failure invalidates Optimized NR protocol

Bug #26913	Three written LCPs during node failure invalidates Optimized NR protocol
Submitted:	7 Mar 2007 10:44	Modified:	25 Apr 2008 8:25
Reporter:	Johan Andersson	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S3 (Non-critical)
Version:	5.1.*	OS:	Any (*)
Assigned to:	Jonas Oreland	CPU Architecture:	Any
Tags:	LCP, Node recovery, NR

Description:
during node failure, if three LCPs have started during the Nodefailure, before the node is recovered, then the node when it will recover cannot make use of the optimized node recovery protocol.

This means that if three LCPs have been started, all data will be resynced from the started ndbd node. This slows down recovery times a lot.

How to repeat:
* start 2 ndbd node cluster (TimeBetweenLocalCheckpoints=20)
* Create the following tables:

create table t1(a char(32), b integer unsigned not null, c char(32), d integer unsigned not null, e varchar(440), primary key(a,b), index (d)) engine=ndb;

create table t2(a char(32), b integer unsigned not null, c char(32), d integer unsigned not null, e varchar(440), primary key(a,b), index (d)) engine=ndb

* Use hugoLoad to load in ~500K records (smaller might work, but this is safe to reproduce):
./hugoLoad  -r 500000 -l 1 -d test t1

Force an LCP to be written:
ndb_mgm>all dump 7099
-> wait for LCP to complete (check cluster log)
-> wait a few seconds for a GCP to also complete
 
* Kill one ndbd node
kill -9 <pid angel ndbd> <pid ndbd>

* restarting the failed ndbd node will recover the node correctly using the optimized node recovery protocol.

* if you have restarted the node (which is just done to verify that the optimized NR protocol is working), then kill it again:
kill -9 <pid angel ndbd> <pid ndbd>

* Load data into table t2 (no changes have been made to t1)
./hugoLoad  -r 500000 -l 1 -d test t2

* Restart failed ndbd node

* It will _not_ restore t1 from its local data, but copy _all_ data from the started node, thus breaking the optimized node recovery protocol.

* Checking the cluster logs indicates that three LCPs have been started before the other ndbd node is recovered.

Suggested fix:
Just do it :)

Decentralize a lot of things.

ALL CLUSTERLOG CHECKPOINT=8
ALL CLUSTERLOG NODERESTART=15

are good things to have written to the cluster log when verifying this.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/37839

ChangeSet@1.2681, 2007-11-15 12:27:35+01:00, jonas@perch.ndb.mysql.com +1 -0
  ndb - bug#26913
    Dont invalidate LCP of nodes that dont participate in LCP

pushed to 51-telco (to be merged with telco-63)

Documented bugfix in 5.1.23-ndb-6.3.8 as follows:

        Under some circumstances, a recovering data node did not use its
        own data, instead copying data from another node even when this
        was not required. This in effect bypassed the optimized node
        recovery protocol and caused recovery times to be unnecessarily
        long.

Left bug in PQ status pending further merges.

no further merges, (only fixed in >= telco-6.3)
closing