Description:
I just wasted more than a day on tracking down a problem on a customer system that lead to node failures with either
Message: File not found (Ndbd file system inconsistency error, please report a bug)
Error: 2815
Error data: DBDICT: File system open failed during FsConnectRecord state 1. OS errno: 2
Error object: DBDICT (Line: 1058) 0x00000002
Program: ndbd
or
Message: File not found (Ndbd file system inconsistency error, please report a bug)
Error: 2815
Error data: DBLQH: File system open failed. OS errno: 2
Error object: DBLQH (Line: 3643) 0x00000002
Program: ndbd
which could only be solved by an --initial node restart.
Only after tracking a failed node restart with strace we were able to identify the actual file that couldn't be opened (turned out someone had gzipped some large FragLog files in the cluster data directory for a yet unknown reason).
If the error log entries had shown the problematic file name or path right away this would probably only have been a matter of less than an hour to track down the root cause :(
How to repeat:
Shut down a data node, rename or remove some file in the ndb_#_fs directory that the node will try to open at startup, see how the ndbd process logs a "File system open failed ... OS Errno: 1" error but does not tell WHICH file
Suggested fix:
Include the file name / path in the error message, or if this leads to problems with the fixed size error entry format: at least log an appropriate message in the nodes .out log file that includes the file path
