Bug #24016 error 6303: 'Unable to find a restorable replica Ndbd file system inconsistency

• Submitted: 6 Nov 2006 15:34
• Modified: 13 Nov 2006 9:33
• Reporter: Kris Buytaert (Candidate Quality Contributor)
• Status: Duplicate
• Category: MySQL Cluster: Cluster (NDB) storage engine
• Severity: S2 (Serious)
• Version: 5.1.11
• Assigned to: Assigned Account
• CPU Architecture: Any
• Tags: ndbd diskbased cluster

[6 Nov 2006 15:34] Kris Buytaert

Description:
ndbd doesn't start anymore with the following error

 Node 4: Forced node shutdown completed. Occured during startphase 4. Initiated by signal 0. Caused by error 6303: 'Unable to find a restorable replica(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial

As a result the other starting node also fails to start 

 Node 3: Forced node shutdown completed. Occured during startphase 4. Initiated by signal 0. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.

Upon trying to query the database one morning all queries came back with 
"cluster failure"  , we shut down the cluster and tried restarting. 
We could only get the cluster back up and running after doing a --initial 

The above errors actually was seen on both nodes.

How to repeat:
We tried restarting multiple times , each time the same errors.

[6 Nov 2006 15:36] Kris Buytaert

ndb log files

Attachment: bug-data-24016.tar.gz (application/x-gzip, text), 169.26 KiB.

[6 Nov 2006 15:43] MySQL Verification Team

Updating Category.

[8 Nov 2006 12:05] Kris Buytaert

The initial crash was due to the Linux OOM Killer shooting in action.
However ndbd recovery should have been possible.
And takeover should have taken place correctly.

[13 Nov 2006 9:33] Jonas Oreland

Hi,

This looks very much like http://bugs.mysql.com/bug.php?id=22892
which was fixed just recently