Bug #47852 Crash after upgrade to 7.0.7
Submitted: 6 Oct 2009 8:38 Modified: 8 Oct 2009 9:42
Reporter: Daniel Herlitz Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:7.0.7 OS:Linux
Assigned to: Jonas Oreland CPU Architecture:Any

[6 Oct 2009 8:38] Daniel Herlitz
Description:
A couple of weeks ago we upgraded one of our systems from 7.0.6 to 7.0.7. Since this is a staging environment we frequently do full database drops + recreates from SQL. Usually no problems. (I mention this in case it might be of any relevance). 

This morning someone decided to reboot the machines and now we can't get NDB up and running again:

2009-10-06 09:58:36 [ndbd] ALERT    -- Node 2: Forced node shutdown completed. Occured during startphase 4. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

Since I noticed similar bug reports regarding ndbmtd I think I should point out that we are running ndbd.

How to repeat:
Don't know

Suggested fix:
Don't know
[6 Oct 2009 8:39] Daniel Herlitz
Config.ini

Attachment: config.ini (application/octet-stream, text), 631 bytes.

[6 Oct 2009 8:39] Daniel Herlitz
ndb_2_error.log

Attachment: ndb_2_error.log (application/octet-stream, text), 1.04 KiB.

[6 Oct 2009 8:39] Daniel Herlitz
ndb_2_trace.log.2

Attachment: ndb_2_trace.log.2.gz (application/x-gzip, text), 37.40 KiB.

[6 Oct 2009 12:39] Gustaf Thorslund
Daniel,

Could you please send the output from the ndb_error_reporter?

/Gustaf
[6 Oct 2009 12:46] Jonas Oreland
FYI: The datanode crashes as head and tail meets in redo-log.
     This is not related to 7.0.7 per se.
     Increasing size of your redo-log
       (using either NoOfFragmentLogFiles or FragmentLogFileSize) should fix it.

Note: Changing size of redo-log, can only be done using either 
      initial-node-restart, or initial cluster start.
[6 Oct 2009 13:43] Daniel Herlitz
error report

Attachment: ndb_error_report_20091006153207.tar.bz2 (application/octet-stream, text), 231.10 KiB.

[6 Oct 2009 13:45] Daniel Herlitz
Output from ndb_error_reporter is attached. However, on the third attempt NDB acutally started for us, so the information might not be of any use to you. But thank you for the tip. Will look at increasing the redo log.
[8 Oct 2009 9:42] Jonas Oreland
given that this is not a 7.0.7 bug
and it's more of a configuration error
i'll close this as not a bug.