MySQL Bugs: #65611: Cannot restart a data node, gives error 2308

Bug #65611	Cannot restart a data node, gives error 2308
Submitted:	14 Jun 2012 6:26	Modified:	25 Aug 2016 11:41
Reporter:	Shaun van Wyngaard	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	7.2.6	OS:	Windows (7-64bit)
Assigned to:		CPU Architecture:	Any
Tags:	cluster 2308 node failed startphase 5

Description:
3 PC's, 1 is mngt (32bit XP), other two each have a a data node + SQL node on them (64bit Windows 7), 8GB RAM each

I noticed one data node had gone down, tried to do a restart of node.
Gets to phase 5 and gives error 2308. I tried restarting it a few times, even with a --initial restart.

Caused by error 2308: 'Another node failed during system 
restart, please investigate error(s) on other node(s)(Restart error). 
Temporary error, restart node'. 

However, from the initial setup of these PCs with their node no.s and IP addresses, the only workaround I've found is to give the PC's new IP addresses, and new node numbers, then do a --initial restart on both nodes, then do a restore of data.

I have tried doing a --initial restart on the node that is down while the other is still up, however it gives the above error. If I re-do the IP and nodes changes, this will be the 3rd time I am doing this which is very inconvenient.
I have learnt to do a baskup on the songle running data node right before I shut it down!

Unfortunately I don't have the ndb_error_reporter utility (because of a Windows install?)

How to repeat:
Not sure, suspect that there was a network connection disruption, or a quick power outage, but not sure.

Suggested fix:
Workaround is to backup, assign new IP's, new nedes, do intial restart, restore data.

I tried to replicate but couldn't reproduce the reported issue. Is this issue still repeatable? Do you have a test case which could reproduce this issue? Please let us know.

Please note that we need much more information than what you have added initially. 
Going forward if you raise any defect then please upload the complete set of logs and trace files. 

## Logs in management, storage nodes are named as below

 ndb_<nodeid>_cluster
 ndb_<nodeid>_out.log
 ndb_<nodeid>_error.log
 ndb_<nodeid>_trace.log.[1..9]

These files contain both regular progress and error information. Preferably using the ndb_error_reporter utility(if not then collect all the
relevant logs from management, storage, config.ini and compress before uploading):

  http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-programs-ndb-error-reporter.html
  
If the data you need to attach is more than 500KB, you should create a compressed archive of the data and a README file that describes the data with a filename that 
includes the bug number (example: bug-data-62607.zip), and use FTP (login with the userid anonymous and your email address) to upload the archive
to ftp://ftp.oracle.com/support/incoming/. Once you have uploaded the file, add a comment to this bug to notify us about it. 
Note: This directory is unlistable, which means that once you have uploaded your file, you will not be able to see it. 
By default, all files will be deleted after 21 days with 2 advanced email warnings.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".