MySQL Bugs: #25984: Eight node restart failures in a row can kill a cluster

Bug #25984	Eight node restart failures in a row can kill a cluster
Submitted:	31 Jan 2007 16:30	Modified:	8 Feb 2007 8:40
Reporter:	Hartmut Holzgraefe
Status:	Closed
Category:	Server: Cluster	Severity:	S3 (Non-critical)
Version:	4.1,5.0,5.1	OS:	Any (*)
Assigned to:	Jonas Oreland	Target Version:

Description:
A node failing during restart eight times in a row will cause other nodes to fail with 

Status: Ndbd file system error, restart node initial
Message: Too many crashed replicas (8 consecutive node restart failures) (Ndbd file system
limit exceeded)
Error: 6300
Error data: DbdihMain.cpp
Error object: DBDIH (Line: 12062) 0x0000000a
Program: /usr/mysql/libexec/ndbd

which brings down the cluster.

How to repeat:
In a working cluster: restart a node and kill it during restart often enough

Suggested fix:
short term: raise the limit from 8 to a higher value

long term: completely remove the limitation

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/19256

ChangeSet@1.2419, 2007-02-02 17:07:15+01:00, jonas@eel.(none) +3 -0
  ndb - bug#25984 - more than 7 failed node restart can cause cluster failure
  new behaviour is as follows:
  1) node is refused to start, and should fail with message in error log that it must be
restarted --initial
  2) if cluster failure in this situation, node must also be restarted --intial
     if not SR will fail, with this message

pushed to 5.1.16

Thank you for your bug report. This issue has been committed to our source repository of
that product and will be incorporated into the next release.

If necessary, you can access the source repository and build the latest available version,
including the bug fix. More information about accessing the source trees is available at

    http://dev.mysql.com/doc/en/installing-source.html

Documented in 5.1.16 changelog.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/21156

ChangeSet@1.2106, 2007-03-05 16:16:01+01:00, jonas@perch.ndb.mysql.com +3 -0
  ndb - wl2325-5.0
    Bug #25984 8 failed node restart kills alive cluster