Bug #26006 No way to detect ACID-validation on cluster restarts
Submitted: 1 Feb 2007 13:44 Modified: 18 Apr 2007 6:16
Reporter: Hartmut Holzgraefe Email Updates:
Status: Won't fix Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:5.0, 5.1 OS:Any (*)
Assigned to: CPU Architecture:Any

[1 Feb 2007 13:44] Hartmut Holzgraefe
Description:
If a whole cluster goes down, e.g. due to severe network problems or a power loss on all data nodes, the most recently commited transactions may be lost as MySQL Cluster is not fully ACID compliant in this case. MySQL Cluster does not flush its REDO logs to disk on each commit (as this would limit transaction rates to more or less one per disk rotation) which is by design and is a documentated limitation. Usually this is not an issue as full cluster failures are rare so the ACID-principle violation itself is not the scope of this bug report.

In the rare case of a full cluster restart there doesn't seem to be a way to distinguish whether some of the most recent transactions where lost on the way or not.

There is obviously no way to tell which (or how many) transactions were lost, but there SHOULD be a way to tell whether the cluster is in a clean ACID state after a system restart or whether it is only "ACId" with the risk of some transactions having been lost between shutdown (or failure) and restart. If in doubt the system should assume that not all transactions made it to the REDO logs.

Things needed here are:

- a clear message in the cluster log stating potential transaction loss
  (this needs to be on a single log line by itself, not something that 
   can be deducted from a sequence of log messages for ease of monitoring
   and log file rotation reasons)

- a configuration option that would make the cluster abort a system restart
  in this case to allow administrators or automatic monitoring solutions
  to become aware of the situation and to be able to intervene *before*
  the cluster becomes operational again

Knowledge of the ACIDity state is crucial if the cluster is a replication slave to a non-cluster master for example.

How to repeat:
Pull the power from all data nodes at the same time for example, then see if you can easily figure out from the logs whether your cluster might have lost some commits or not ...

Suggested fix:
- have nodes write a "clean shutdown" marker to the end of their REDO log if they 
  were able to flush all pending commited transactions to disk before going down

- let nodes check for the "clean shutdown" marker first if REDO log processing is   
  needed during system restart

- if log replay on system startup would require processing of REDO logs without   
  "clean shotdown" marker at the very end -> log "possible loss of most recent
  transactions on system restart", terminate restart if requested by the new
  configuration option

- on successfull REDO log processing and startup -> write a new marker to the 
  REDO log indicating that it is *not* in the "clean shutdown" start anymore
  and make sure it is flushed to disk before resuming operation
[18 Apr 2007 6:16] Tomas Ulin
This will not be fixed.

Fix for feature request 26783, will function equally well (even better) for the purpose of finding out status on slave side after shutdown/failure.

BR,

Tomas Ulin