Bug #25924 Error 2311: all data nodes crashed - can not restart any data node
Submitted: 29 Jan 2007 17:51 Modified: 16 Mar 2008 14:53
Reporter: Bugs System Email Updates:
Status: No Feedback Impact on me:
Category:MySQL Cluster: Disk Data Severity:S1 (Critical)
Version:5.1.14 OS:Linux (CentOS4.4 x86_64)
Assigned to: CPU Architecture:Any
Tags: cluster, crash, data, restart

[29 Jan 2007 17:51] Bugs System
4 node, 2 replica, mysql 5.1.14 disk based cluster was running for about a week problem free. Came in from the weekend to all data nodes crashed, and all mysqld nodes showing as not connected to the cluster. mysqld was running and accessible on each node though. Trying to do a ndbd returns error 2311, and doing an ndbd --initial does the same. Only thing allowing a start of all ndbd nodes is a delete of all the data and undo files, doing an --initial restart, and recreating all the data.

How to repeat:
Start a 4 node, 2 replica cluster, wait 1 week.
[29 Jan 2007 18:02] Jonas Oreland

We need logfiles, try using ndb_error_reporter

[29 Jan 2007 18:07] Bugs System
The following two files containing full error, trace, out, and config files have been uploaded to ftp.mysql.org:/pub/mysql/upload/

[31 Jan 2007 16:17] Bugs System
Updated OS
[31 Jan 2007 16:59] Bugs System
These are the exact MySQL provided RPM's used on the nodes (management rpm's only included on first node):

[16 Feb 2008 14:53] Valeriy Kravchuk
Please, try to repeat with a newer version, 5.1.23-rc, and inform about the results.
[17 Mar 2008 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".