Bug #54688 Ndbd file system inconsistency error, please report a bug
Submitted: 22 Jun 2010 9:03 Modified: 6 Aug 2010 8:48
Reporter: lee zhenhua Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql version 5.1.35 ,ndb-7.0.7 OS:Other (centos 5.2 x86_64)
Assigned to: CPU Architecture:Any
Tags: Caused by error 2310: 'Error while reading the REDO log

[22 Jun 2010 9:03] lee zhenhua
Description:
 hi
  I use "start backup " backup data  in mgm node .Then I deleted database, and shutdown all nodes.
    start MGM node
    use ndbd --initial start data node . Then Restore data all node.  Then i restart the MGM DATA node .
In the startup times the mistake

ndb_mgm> Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
Node 5: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
Node 6: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.

MGM node error messages
2010-06-22 16:16:58 [MgmSrvr] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-22 16:16:58 [MgmSrvr] ALERT    -- Node 1: Node 4 Disconnected
2010-06-22 16:16:59 [MgmSrvr] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-22 16:16:59 [MgmSrvr] ALERT    -- Node 1: Node 3 Disconnected
2010-06-22 16:17:00 [MgmSrvr] ALERT    -- Node 5: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2010-06-22 16:17:00 [MgmSrvr] ALERT    -- Node 6: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2010-06-22 16:17:00 [MgmSrvr] ALERT    -- Node 1: Node 5 Disconnected
2010-06-22 16:17:00 [MgmSrvr] ALERT    -- Node 1: Node 6 Disconnected

How to repeat:
ndbd node error message
Time: Tuesday 22 June 2010 - 16:16:58
Status: Ndbd file system error, restart node initial
Message: Error while reading the REDO log (Ndbd file system inconsistency error, please report a bug)
Error: 2310
Error data: Error while reading REDO log. from 16462
part: 3 D=11, F=1 Mb=159 FP=5119 W1=8185 W2=7 : end of log wo/ having found last GCI gci: 920
Error object: DBLQH (Line: 16570) 0x0000000a
Program: ndbd
Pid: 11922
Trace: /mysql/var/ndb_3_trace.log.3
Version: mysql-5.1.35 ndb-7.0.7
***EOM**
next  is  ndb_3_trace.log.3 log

--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402235 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402234 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402234 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402233 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402233 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402232 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402232 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402231 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402231 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402230 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402230 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402229 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402229 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402228 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402228 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402227 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402227 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402226 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402226 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402224 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 248 "DBACC", r.proc: 3, r.sigId: 108402225 gsn: 253 "EXPANDCHECK2" prio: 1
s.bn: 248 "DBACC", s.proc: 3, s.sigId: 108402224 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'000001d1 H'00004a52 H'00007fff
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402224 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402223 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402223 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402222 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402222 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402221 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402221 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402220 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402220 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402219 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000000 H'00000000

Suggested fix:
I according to clew restart.
[22 Jun 2010 13:52] lee zhenhua
use perror dispaly 
[root@MGMM logs]# perror --ndb 2310
NDB error code 2310: Error while reading the REDO log: Ndbd file system error, restart node initial: Ndbd file system inconsistency error, please report a bug
[24 Jun 2010 7:12] lee zhenhua
All data nodes errors are the same。 as follows

RESTORE table: 4 0 rows applied
RESTORE table: 4 0 rows applied
RESTORE table: 5 0 rows applied
RESTORE table: 5 0 rows applied
RESTORE table: 6 0 rows applied
RESTORE table: 6 0 rows applied
RESTORE table: 7 1930900 rows applied
RESTORE table: 7 1932495 rows applied
records: 16 len: 170 left: 248
RESTORE table: 8 571493 rows applied
records: 16 len: 164 left: 272
RESTORE table: 8 573665 rows applied
RESTORE table: 9 2745486 rows applied
RESTORE table: 9 2748543 rows applied
RESTORE table: 10 1196712 rows applied
RESTORE table: 10 1200062 rows applied
RESTORE table: 11 2743080 rows applied
RESTORE table: 11 2746158 rows applied
RESTORE table: 12 572936 rows applied
RESTORE table: 12 571474 rows applied
2010-06-24 14:59:52 [ndbd] INFO     -- Error while reading REDO log. from 16462
part: 0 D=8, F=4 Mb=218 FP=7007 W1=8158 W2=7 : end of log wo/ having found last GCI gci: 213
2010-06-24 14:59:52 [ndbd] INFO     -- DBLQH (Line: 16570) 0x0000000a
2010-06-24 14:59:52 [ndbd] INFO     -- Error handler startup shutting down system
2010-06-24 14:59:52 [ndbd] INFO     -- Error handler shutdown completed - exiting
2010-06-24 14:59:52 [ndbd] INFO     -- Angel received ndbd startup failure count 1.
2010-06-24 14:59:53 [ndbd] ALERT    -- Node 4: Forced node shutdown completed. Occured during startphase 4. Caused 

by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd 

file system error, restart node initial'.
[6 Jul 2010 8:36] Andrew Hutchings
There have been several bug fixes related to this error since 7.0.7.  Please try updating to the latest 7.0 release to see if this fixes the problem.
[6 Aug 2010 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".