Bug #74438 ndbd cannot be started caused by error 2807

• Submitted: 18 Oct 2014 16:42
• Modified: 2 Oct 2015 15:01
• Reporter: Daniel Yuan
• Status: Not a Bug
• Category: MySQL Cluster: Cluster (NDB) storage engine
• Severity: S1 (Critical)
• Version: 7.3.6
• OS: Linux (Red Hat(64-bit))
• Assigned to: MySQL Verification Team
• CPU Architecture: Any
• Tags: cannot started, caused by error 2807, MySQL Cluster, ndbd

[18 Oct 2014 16:42] Daniel Yuan

Description:
I installed MySQL Cluster 7.3.6 in the virtual machines of my laptop. Several days ago my laptop shutdown without shutdown VMs in advance. Then after I restarted my laptop and opened virtual machines, ndb_mgmd can be started but ndbd cannot be started successfully with the following error message in cluster log: 

2014-10-17 22:04:13 [MgmtSrvr] INFO     -- Node 1: Node: 1 StartLog: [GCI Keep: 316100 LastCompleted: 324090 NewestRestorable: 324090]
2014-10-17 22:04:20 [MgmtSrvr] INFO     -- Node 1: Applying undo to LCP: 243
2014-10-17 22:04:20 [MgmtSrvr] INFO     -- Node 1: Undo head - undo2.log page: 35 lsn: 3504585
2014-10-17 22:04:20 [MgmtSrvr] INFO     -- Node 1:    - next - undo3.log(2816535)
2014-10-17 22:04:20 [MgmtSrvr] INFO     -- Node 1:    - next - undo1.log(2811017)
2014-10-17 22:04:26 [MgmtSrvr] INFO     -- Node 1: Logfile group: 10 
2014-10-17 22:04:26 [MgmtSrvr] INFO     -- Node 1:   head: undo2.log page: 35
2014-10-17 22:04:26 [MgmtSrvr] INFO     -- Node 1:   tail: undo2.log page: 32
2014-10-17 22:04:26 [MgmtSrvr] INFO     -- Node 1: Flushing page cache after undo completion
2014-10-17 22:04:26 [MgmtSrvr] INFO     -- Node 1: Flushing complete
2014-10-17 22:04:32 [MgmtSrvr] ALERT    -- Node 1: Forced node shutdown completed. Occured during startphase 4. Caused by error 2807: 'File has already been opened(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2014-10-17 22:04:32 [MgmtSrvr] ALERT    -- Node 49: Node 1 Disconnected

 

How to repeat:
I restarted VMs and tried to restart ndbd from different data node several times. Each time I got the same error message.

[2 Oct 2015 15:01] MySQL Verification Team

Hi,

This is not a bug. Your ndbcluster filesystem got corrupted and data node can't start. To solve this problem normal procedure is to start cluster without this corrupt node (use --no-wait) and then when cluster starts (other nodes) you start this corrupt node with --initial so it clears up local ndbcluster filesystem and fetches the data from replica node on the cluster

kind regards
Bogdan Kecman