Bug #34026 Rebooting NDB node gives configuration changed message --initial dies at phase 5
Submitted: 23 Jan 2008 23:51 Modified: 10 May 2008 20:53
Reporter: Brett Trotter Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:5.1.22 OS:Linux (CentOS 5.1)
Assigned to: CPU Architecture:Any

[23 Jan 2008 23:51] Brett Trotter
Description:
Rebooted an NDB node, refused to start with the configuration changed error (despite having not changed config.ini on the server or any of the my.cnf's).

So, I did an ndbd --initial out of /var/lib/mysql-cluster and on ndb_mgm:

Node 3: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

on ndb_3_error.log:
Current byte-offset of file-pointer is: 568

Time: Wednesday 23 January 2008 - 17:30:42
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug
)
Error: 2341
Error data: dbdict/Dbdict.cpp
Error object: DBDICT (Line: 3126) 0x0000000e
Program: ndbd
Pid: 3354
Trace: /var/lib/mysql-cluster/ndb_3_trace.log.1
Version: Version 5.1.22 (rc)
***EOM***

Tried rm -rf-ing the whole /var/lib/mysql-cluster/* and redoing ndbd --initial and got the same error.

ndb_3_out.log has
2008-01-23 17:30:17 [ndbd] INFO     -- Angel pid: 3353 ndb pid: 3354
2008-01-23 17:30:17 [ndbd] INFO     -- NDB Cluster -- DB node 3
2008-01-23 17:30:17 [ndbd] INFO     -- Version 5.1.22 (rc) --
2008-01-23 17:30:17 [ndbd] INFO     -- Configuration fetched at 10.0.130.12 port 1186
2008-01-23 17:30:17 [ndbd] INFO     -- Start initiated (version 5.1.22)
2008-01-23 17:30:17 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 20Mb initial: 20Mb
WOPool::init(61, 9)
RWPool::init(82, 13)
RWPool::init(a2, 18)
RWPool::init(c2, 13)
RWPool::init(122, 18)
RWPool::init(142, 15)
WOPool::init(41, 8)
RWPool::init(e2, 12)
RWPool::init(102, 51)
WOPool::init(21, 6)
2008-01-23 17:30:42 [ndbd] INFO     -- dbdict/Dbdict.cpp
2008-01-23 17:30:42 [ndbd] INFO     -- DBDICT (Line: 3126) 0x0000000e
2008-01-23 17:30:42 [ndbd] INFO     -- Error handler startup shutting down system
2008-01-23 17:30:42 [ndbd] INFO     -- Error handler shutdown completed - exiting
2008-01-23 17:30:42 [ndbd] INFO     -- Angel received ndbd startup failure count 1.
2008-01-23 17:30:42 [ndbd] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 5. Caused by error 2341: 'I
nternal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporar
y error, restart node'.

As trace log is huge, I'll attach it momentarily instead of pasting.

How to repeat:
Run ndbd --initial in an existing 2-node cluster.
[26 Jan 2008 10:59] Jonas Oreland
Hi,

can you upload a tarball with D1/DBDICT/* from a living node 
and also attach the output of "ndb_show_tables"

/Jonas
[10 Apr 2008 20:53] Sveta Smirnova
Brett,

could you please provide information which Jonas requested.
[10 May 2008 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".