Description:
Rebooted an NDB node, refused to start with the configuration changed error (despite having not changed config.ini on the server or any of the my.cnf's).
So, I did an ndbd --initial out of /var/lib/mysql-cluster and on ndb_mgm:
Node 3: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
on ndb_3_error.log:
Current byte-offset of file-pointer is: 568
Time: Wednesday 23 January 2008 - 17:30:42
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug
)
Error: 2341
Error data: dbdict/Dbdict.cpp
Error object: DBDICT (Line: 3126) 0x0000000e
Program: ndbd
Pid: 3354
Trace: /var/lib/mysql-cluster/ndb_3_trace.log.1
Version: Version 5.1.22 (rc)
***EOM***
Tried rm -rf-ing the whole /var/lib/mysql-cluster/* and redoing ndbd --initial and got the same error.
ndb_3_out.log has
2008-01-23 17:30:17 [ndbd] INFO -- Angel pid: 3353 ndb pid: 3354
2008-01-23 17:30:17 [ndbd] INFO -- NDB Cluster -- DB node 3
2008-01-23 17:30:17 [ndbd] INFO -- Version 5.1.22 (rc) --
2008-01-23 17:30:17 [ndbd] INFO -- Configuration fetched at 10.0.130.12 port 1186
2008-01-23 17:30:17 [ndbd] INFO -- Start initiated (version 5.1.22)
2008-01-23 17:30:17 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 20Mb initial: 20Mb
WOPool::init(61, 9)
RWPool::init(82, 13)
RWPool::init(a2, 18)
RWPool::init(c2, 13)
RWPool::init(122, 18)
RWPool::init(142, 15)
WOPool::init(41, 8)
RWPool::init(e2, 12)
RWPool::init(102, 51)
WOPool::init(21, 6)
2008-01-23 17:30:42 [ndbd] INFO -- dbdict/Dbdict.cpp
2008-01-23 17:30:42 [ndbd] INFO -- DBDICT (Line: 3126) 0x0000000e
2008-01-23 17:30:42 [ndbd] INFO -- Error handler startup shutting down system
2008-01-23 17:30:42 [ndbd] INFO -- Error handler shutdown completed - exiting
2008-01-23 17:30:42 [ndbd] INFO -- Angel received ndbd startup failure count 1.
2008-01-23 17:30:42 [ndbd] ALERT -- Node 3: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'I
nternal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporar
y error, restart node'.
As trace log is huge, I'll attach it momentarily instead of pasting.
How to repeat:
Run ndbd --initial in an existing 2-node cluster.