Description:
hi
I use "start backup " backup data in mgm node .Then I deleted database, and shutdown all nodes.
start MGM node
use ndbd --initial start data node . Then Restore data all node. Then i restart the MGM DATA node .
In the startup times the mistake
ndb_mgm> Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
Node 5: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
Node 6: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
MGM node error messages
2010-06-22 16:16:58 [MgmSrvr] ALERT -- Node 4: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-22 16:16:58 [MgmSrvr] ALERT -- Node 1: Node 4 Disconnected
2010-06-22 16:16:59 [MgmSrvr] ALERT -- Node 3: Forced node shutdown completed. Occured during startphase 4. Caused by error 2310: 'Error while reading the REDO log(Ndbd file system inconsistency error, please report a bug). Ndbd file system error, restart node initial'.
2010-06-22 16:16:59 [MgmSrvr] ALERT -- Node 1: Node 3 Disconnected
2010-06-22 16:17:00 [MgmSrvr] ALERT -- Node 5: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2010-06-22 16:17:00 [MgmSrvr] ALERT -- Node 6: Forced node shutdown completed. Occured during startphase 4. Caused by error 2308: 'Another node failed during system restart, please investigate error(s) on other node(s)(Restart error). Temporary error, restart node'.
2010-06-22 16:17:00 [MgmSrvr] ALERT -- Node 1: Node 5 Disconnected
2010-06-22 16:17:00 [MgmSrvr] ALERT -- Node 1: Node 6 Disconnected
How to repeat:
ndbd node error message
Time: Tuesday 22 June 2010 - 16:16:58
Status: Ndbd file system error, restart node initial
Message: Error while reading the REDO log (Ndbd file system inconsistency error, please report a bug)
Error: 2310
Error data: Error while reading REDO log. from 16462
part: 3 D=11, F=1 Mb=159 FP=5119 W1=8185 W2=7 : end of log wo/ having found last GCI gci: 920
Error object: DBLQH (Line: 16570) 0x0000000a
Program: ndbd
Pid: 11922
Trace: /mysql/var/ndb_3_trace.log.3
Version: mysql-5.1.35 ndb-7.0.7
***EOM**
next is ndb_3_trace.log.3 log
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402235 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402234 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402234 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402233 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402233 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402232 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402232 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402231 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402231 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402230 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402230 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402229 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402229 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402228 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402228 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402227 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402227 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402226 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402226 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402224 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 248 "DBACC", r.proc: 3, r.sigId: 108402225 gsn: 253 "EXPANDCHECK2" prio: 1
s.bn: 248 "DBACC", s.proc: 3, s.sigId: 108402224 length: 3 trace: 0 #sec: 0 fragInf: 0
H'000001d1 H'00004a52 H'00007fff
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402224 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402223 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402223 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402222 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402222 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402221 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402221 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402220 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
--------------- Signal ----------------
r.bn: 262 "RESTORE", r.proc: 3, r.sigId: 108402220 gsn: 164 "CONTINUEB" prio: 1
s.bn: 262 "RESTORE", s.proc: 3, s.sigId: 108402219 length: 2 trace: 0 #sec: 0 fragInf: 0
H'00000000 H'00000000
Suggested fix:
I according to clew restart.