MySQL Bugs: #54688: Ndbd file system inconsistency error, please report a bug

Bug #54688	Ndbd file system inconsistency error, please report a bug
Submitted:	22 Jun 2010 9:03	Modified:	6 Aug 2010 8:48
Reporter:	lee zhenhua	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	mysql version 5.1.35 ,ndb-7.0.7	OS:	Other (centos 5.2 x86_64)
Assigned to:		CPU Architecture:	Any
Tags:	Caused by error 2310: 'Error while reading the REDO log

Description:
 hi
  I use "start backup " backup data  in mgm node .Then I deleted database, and shutdown all nodes.
    start MGM node
    use ndbd --initial start data node . Then Restore data all node.  Then i restart the MGM DATA node .
In the startup times the mistake

ndb_mgm> Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
Node 5: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
Node 6: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.

MGM node error messages
2010-06-22 16:16:58 [MgmSrvr] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-22 16:16:58 [MgmSrvr] ALERT    -- Node 1: Node 4 Disconnected
2010-06-22 16:16:59 [MgmSrvr] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-22 16:16:59 [MgmSrvr] ALERT    -- Node 1: Node 3 Disconnected
2010-06-22 16:17:00 [MgmSrvr] ALERT    -- Node 5: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2010-06-22 16:17:00 [MgmSrvr] ALERT    -- Node 6: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2010-06-22 16:17:00 [MgmSrvr] ALERT    -- Node 1: Node 5 Disconnected
2010-06-22 16:17:00 [MgmSrvr] ALERT    -- Node 1: Node 6 Disconnected

How to repeat:
ndbd node error message
Time: Tuesday 22 June 2010 - 16:16:58
Status: Ndbd file system error, restart node initial
Message: Error while reading the REDO log (Ndbd file system inconsistency error, please report a bug)
Error: 2310
Error data: Error while reading REDO log. from 16462
part: 3 D=11, F=1 Mb=159 FP=5119 W1=8185 W2=7 : end of log wo/ having found last GCI gci: 920
Error object: DBLQH (Line: 16570) 0x0000000a
Program: ndbd
Pid: 11922
Trace: /mysql/var/ndb_3_trace.log.3
Version: mysql-5.1.35 ndb-7.0.7
***EOM**
next  is  ndb_3_trace.log.3 log

--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402235 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402234 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402234 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402233 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402233 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402232 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402232 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402231 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402231 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402230 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402230 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402229 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402229 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402228 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402228 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402227 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402227 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402226 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402226 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402224 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 248 "DBACC", r.proc: 3, r.sigId: 108402225 gsn: 253 "EXPANDCHECK2" prio: 1
s.bn: 248 "DBACC", s.proc: 3, s.sigId: 108402224 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'000001d1 H'00004a52 H'00007fff
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402224 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402223 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402223 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402222 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402222 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402221 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402221 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402220 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402220 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402219 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000

Suggested fix:
I according to clew restart.

use perror dispaly 
[root@MGMM logs]# perror --ndb 2310
NDB error code 2310: Error while reading the REDO log: Ndbd file system error, restart node initial: Ndbd file system inconsistency error, please report a bug

All data nodes errors are the same。 as follows

RESTORE table: 4 0 rows applied
RESTORE table: 4 0 rows applied
RESTORE table: 5 0 rows applied
RESTORE table: 5 0 rows applied
RESTORE table: 6 0 rows applied
RESTORE table: 6 0 rows applied
RESTORE table: 7 1930900 rows applied
RESTORE table: 7 1932495 rows applied
records: 16 len: 170 left: 248
RESTORE table: 8 571493 rows applied
records: 16 len: 164 left: 272
RESTORE table: 8 573665 rows applied
RESTORE table: 9 2745486 rows applied
RESTORE table: 9 2748543 rows applied
RESTORE table: 10 1196712 rows applied
RESTORE table: 10 1200062 rows applied
RESTORE table: 11 2743080 rows applied
RESTORE table: 11 2746158 rows applied
RESTORE table: 12 572936 rows applied
RESTORE table: 12 571474 rows applied
2010-06-24 14:59:52 [ndbd] INFO     -- Error while reading REDO log. from 16462
part: 0 D=8, F=4 Mb=218 FP=7007 W1=8158 W2=7 : end of log wo/ having found last GCI gci: 213
2010-06-24 14:59:52 [ndbd] INFO     -- DBLQH (Line: 16570) 0x0000000a
2010-06-24 14:59:52 [ndbd] INFO     -- Error handler startup shutting down system
2010-06-24 14:59:52 [ndbd] INFO     -- Error handler shutdown completed - exiting
2010-06-24 14:59:52 [ndbd] INFO     -- Angel received ndbd startup failure count 1.
2010-06-24 14:59:53 [ndbd] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 4. Caused 

by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd 

file system error, restart node initial'.

There have been several bug fixes related to this error since 7.0.7.  Please try updating to the latest 7.0 release to see if this fixes the problem.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".