MySQL Bugs: #25924: Error 2311: all data nodes crashed

Bug #25924	Error 2311: all data nodes crashed - can not restart any data node
Submitted:	29 Jan 2007 17:51	Modified:	16 Mar 2008 14:53
Reporter:	Bugs System	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Disk Data	Severity:	S1 (Critical)
Version:	5.1.14	OS:	Linux (CentOS4.4 x86_64)
Assigned to:		CPU Architecture:	Any
Tags:	cluster, crash, data, restart

Description:
4 node, 2 replica, mysql 5.1.14 disk based cluster was running for about a week problem free. Came in from the weekend to all data nodes crashed, and all mysqld nodes showing as not connected to the cluster. mysqld was running and accessible on each node though. Trying to do a ndbd returns error 2311, and doing an ndbd --initial does the same. Only thing allowing a start of all ndbd nodes is a delete of all the data and undo files, doing an --initial restart, and recreating all the data.

How to repeat:
Start a 4 node, 2 replica cluster, wait 1 week.

Hi,

We need logfiles, try using ndb_error_reporter

/Jonas

The following two files containing full error, trace, out, and config files have been uploaded to ftp.mysql.org:/pub/mysql/upload/

bug-data-25924.tar.gz
bug-data-25924.readme

Updated OS

These are the exact MySQL provided RPM's used on the nodes (management rpm's only included on first node):

MySQL-ndb-storage-5.1.14-0.glibc23
MySQL-ndb-tools-5.1.14-0.glibc23
MySQL-client-5.1.14-0.glibc23
MySQL-ndb-management-5.1.14-0.glibc23
MySQL-server-5.1.14-0.glibc23
MySQL-ndb-extra-5.1.14-0.glibc23
perl-DBD-MySQL-2.9004-3.1

Please, try to repeat with a newer version, 5.1.23-rc, and inform about the results.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".