MySQL Bugs: #88695: Ndbd file system error, restart node initial

Bug #88695	Ndbd file system error, restart node initial
Submitted:	29 Nov 2017 10:56	Modified:	4 Dec 2017 16:12
Reporter:	Eranga Amarakoon	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Cluster: plugin	Severity:	S2 (Serious)
Version:	5.1.24 (rc)	OS:	CentOS (2.6.18-8.el5PAE)
Assigned to:	MySQL Verification Team	CPU Architecture:	Any

Description:
I have 2 node ndb cluster.

ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=2 (not connected, accepting connect from 10.127.1.3)
id=3 (not connected, accepting connect from 10.127.1.4)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @10.127.1.5  (Version: 5.1.24)

[mysqld(API)]   2 node(s)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)

Due to power surge 10.127.1.4 (NODE 3) Crashed and can not recover the Hard Disk.

10.127.1.3 (NODE 2)'s OS is up and running but the ndb storage is not starting

This is the ndbd_2_error.log

Time: Tuesday 28 November 2017 - 09:26:42
Status: Ndbd file system error, restart node initial
Message: Error while reading the REDO log (Ndbd file system inconsistency error, please report a bug)
Error: 2310
Error data: Error while reading REDO log. from 15380
D=10, F=13 Mb=4 FP=156 W1=2444 W2=128358638 : Invalid logword
Error object: DBLQH (Line: 15418) 0x0000000a
Program: /usr/local/libexec/ndbd
Pid: 20554
Trace: /var/lib/mysql-cluster/ndb_2_trace.log.24
Version: Version 5.1.24 (rc)

I will attach the ndb_2_trace.log.24

How to repeat:
Each time trying to start the node 2

ndb_2_trace.log.24

Attachment: ndb_2_trace.log.24 (application/octet-stream, text), 942.65 KiB.

Hi,

This is not a bug. Due to hardware error your node corrupted the filesystem. 

You have double data with 2 nodes, if both copies are corrupted you should look for backup. Contact MySQL Support if you need help with recovery.

best regards
Bogdan