Description:
If a whole cluster goes down, e.g. due to severe network problems or a power loss on all data nodes, the most recently commited transactions may be lost as MySQL Cluster is not fully ACID compliant in this case. MySQL Cluster does not flush its REDO logs to disk on each commit (as this would limit transaction rates to more or less one per disk rotation) which is by design and is a documentated limitation. Usually this is not an issue as full cluster failures are rare so the ACID-principle violation itself is not the scope of this bug report.
In the rare case of a full cluster restart there doesn't seem to be a way to distinguish whether some of the most recent transactions where lost on the way or not.
There is obviously no way to tell which (or how many) transactions were lost, but there SHOULD be a way to tell whether the cluster is in a clean ACID state after a system restart or whether it is only "ACId" with the risk of some transactions having been lost between shutdown (or failure) and restart. If in doubt the system should assume that not all transactions made it to the REDO logs.
Things needed here are:
- a clear message in the cluster log stating potential transaction loss
(this needs to be on a single log line by itself, not something that
can be deducted from a sequence of log messages for ease of monitoring
and log file rotation reasons)
- a configuration option that would make the cluster abort a system restart
in this case to allow administrators or automatic monitoring solutions
to become aware of the situation and to be able to intervene *before*
the cluster becomes operational again
Knowledge of the ACIDity state is crucial if the cluster is a replication slave to a non-cluster master for example.
How to repeat:
Pull the power from all data nodes at the same time for example, then see if you can easily figure out from the logs whether your cluster might have lost some commits or not ...
Suggested fix:
- have nodes write a "clean shutdown" marker to the end of their REDO log if they
were able to flush all pending commited transactions to disk before going down
- let nodes check for the "clean shutdown" marker first if REDO log processing is
needed during system restart
- if log replay on system startup would require processing of REDO logs without
"clean shotdown" marker at the very end -> log "possible loss of most recent
transactions on system restart", terminate restart if requested by the new
configuration option
- on successfull REDO log processing and startup -> write a new marker to the
REDO log indicating that it is *not* in the "clean shutdown" start anymore
and make sure it is flushed to disk before resuming operation
