Bug #25924 Error 2311: all data nodes crashed - can not restart any data node
Submitted: 29 Jan 2007 17:51 Modified: 16 Mar 2008 14:53
Reporter: Bugs System Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Disk Data Severity:S1 (Critical)
Version:5.1.14 OS:Linux (CentOS4.4 x86_64)
Assigned to: CPU Architecture:Any
Tags: cluster, crash, data, restart

[29 Jan 2007 17:51] Bugs System
Description:
4 node, 2 replica, mysql 5.1.14 disk based cluster was running for about a week problem free. Came in from the weekend to all data nodes crashed, and all mysqld nodes showing as not connected to the cluster. mysqld was running and accessible on each node though. Trying to do a ndbd returns error 2311, and doing an ndbd --initial does the same. Only thing allowing a start of all ndbd nodes is a delete of all the data and undo files, doing an --initial restart, and recreating all the data.

How to repeat:
Start a 4 node, 2 replica cluster, wait 1 week.
[29 Jan 2007 18:02] Jonas Oreland
Hi,

We need logfiles, try using ndb_error_reporter

/Jonas
[29 Jan 2007 18:07] Bugs System
The following two files containing full error, trace, out, and config files have been uploaded to ftp.mysql.org:/pub/mysql/upload/

bug-data-25924.tar.gz
bug-data-25924.readme
[31 Jan 2007 16:17] Bugs System
Updated OS
[31 Jan 2007 16:59] Bugs System
These are the exact MySQL provided RPM's used on the nodes (management rpm's only included on first node):

MySQL-ndb-storage-5.1.14-0.glibc23
MySQL-ndb-tools-5.1.14-0.glibc23
MySQL-client-5.1.14-0.glibc23
MySQL-ndb-management-5.1.14-0.glibc23
MySQL-server-5.1.14-0.glibc23
MySQL-ndb-extra-5.1.14-0.glibc23
perl-DBD-MySQL-2.9004-3.1
[16 Feb 2008 14:53] Valeriy Kravchuk
Please, try to repeat with a newer version, 5.1.23-rc, and inform about the results.
[17 Mar 2008 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".