MySQL Bugs: #54226: Can't force graceful node STOP that would bring down the cluster

Bug #54226	Can't force graceful node STOP that would bring down the cluster
Submitted:	4 Jun 2010 8:31	Modified:	30 Sep 2010 14:11
Reporter:	Hartmut Holzgraefe	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S4 (Feature request)
Version:	mysql-5.1-telco-7.0	OS:	Any
Assigned to:	Magnus Blåudd	CPU Architecture:	Any

Description:
Currently there is no easy way to say 

  "Shut down this node gracefully, even if that would bring down the cluster"

There is "STOP -a", this seems to act more like killing the node process, and so the last committed transactions may be lost.

There are situations where stopping a node is inevitable, even at the cost of bringing down the whole cluster, e.g. in a machines shutdown scripts.

Right now the only way to handle this without creating a MGMAPI application is
to parse the result of "$id STOP" for the "Node shutdown would cause system crash" error message and fall back to a SHUTDOWN ALL in the shutdown script.

How to repeat:
Try to STOP the last remaining node in a node group while there is ongoing DML transaction activity:

 * a simple STOP will be refused as it would bring down the cluster
 
 * "STOP -a" will cause loss of some committed transactions

Suggested fix:
Provide e.g. a "STOP -f" option to force a node stop, even if that requires a cluster shutdown.

Check for NODE_SHUTDOWN_WOULD_CAUSE_SYSTEM_CRASH error code on failed STOP attempt, issue SHUTDOWN ALL instead in this case if --force option is set

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/119356

Pushed into mysql-5.1-telco-7.0 5.1.47-ndb-7.0.19 (revid:magnus.blaudd@sun.com-20100929085447-dz94ehufsz381j09) (version source revid:magnus.blaudd@sun.com-20100929085447-dz94ehufsz381j09) (merge vers: 5.1.47-ndb-7.0.19) (pib:21)

and 7.1.8

Documented in the NDB-7.0.19 and 7.1.8 changelogs as follows:

        It is now possible using the ndb_mgm management client or the
        MGM API to force a data node shutdown or restart even if this
        would force the shutdown or restart of the entire cluster.

        In the management client, this is implemented through the
        addition of the -F (force) option to the STOP and RESTART
        commands. For more information, see
        "Commands in the MySQL Cluster Management Client".

        The MGM API also adds two new methods for forcing such a node
        shutdown or restart; see "ndb_mgm_stop4()",
        and "ndb_mgm_restart4()", for more information
        about these methods.

Also updated the indicated sections of the Manual with the new options/functions.

Closed.