Description:
RONDB-1114: Do not start LCP fragment checkpoints before the LCP round
When an LCP participant fails, execNODE_FAILREP calls startNextChkpt
so that the LCP does not hang waiting for the failed node. The guard
only checked that no LCP master takeover was in progress.
m_participatingLQH is set at the very start of the LCP, before
START_LCP_REQ is sent, so the guard was also satisfied when a
participant failed while the master was still waiting for its
START_LCP_CONF. The master then sent LCP_FRAG_ORD to the surviving
LQHs from the node failure handling. The same node failure handling
queues a START_LCP_CONF for the failed node, which completes the
handshake and releases the start LCP mutex, and startLcpRoundLoopLab
then failed ndbrequire(noOfStartedChkpt == 0) for the first live
node, error 2341 in the master.
This only happens on a master that has not yet run an LCP round,
typically a master that has just taken over: a master that has
completed a round still has all participants in
m_allReplicasQueuedLQH, so the premature call sends nothing.
Require c_lcp_runs_with_pause_support as well. It is set when the
master has passed the START_LCP_REQ step, right before the LCP round
starts, both in the normal LCP start and in LCP master takeover, and
it is cleared when the LCP completes. Before the round there is
nothing to drive: the round starts with the reduced set of
participants once the handshake completes. Assert that the start LCP
mutex has been released when the call is made.
Reproduced with testNodeRestart -n LcpStartParticipantFail.
How to repeat:
See above
Suggested fix:
+ *
+ * It must also not be called before the master has entered the LCP
+ * round. m_participatingLQH is set at the very start of the LCP,
+ * before START_LCP_REQ is sent, so a participant can fail while the
+ * master is still waiting for START_LCP_CONF. Starting fragment
+ * checkpoints here in that case makes startLcpRoundLoopLab, which is
+ * reached through the START_LCP_CONF queued above for the failed
+ * node and the asynchronous release of the start LCP mutex, fail
+ * ndbrequire(noOfStartedChkpt == 0). There is nothing to drive
+ * here in that case either: the round starts with the reduced set
+ * of participants once the handshake completes.
+ *
+ * c_lcp_runs_with_pause_support is set when the master has passed
+ * the START_LCP_REQ step, right before the LCP round starts, both
+ * in the normal LCP start and in LCP master takeover, and it is
+ * cleared when the LCP completes.
*/
- if (check_more_start_lcp && c_lcpMasterTakeOverState.state == LMTOS_IDLE) {
+ if (check_more_start_lcp && c_lcpMasterTakeOverState.state == LMTOS_IDLE &&
+ c_lcp_runs_with_pause_support) {
jam();
ndbrequire(isMaster());
+ /* The start LCP mutex is released before the LCP round starts */
+ ndbassert(c_startLcpMutexHandle.isNull());
startNextChkpt(signal);
}
} // Dbdih::execNODE_FAILREP()
Description: RONDB-1114: Do not start LCP fragment checkpoints before the LCP round When an LCP participant fails, execNODE_FAILREP calls startNextChkpt so that the LCP does not hang waiting for the failed node. The guard only checked that no LCP master takeover was in progress. m_participatingLQH is set at the very start of the LCP, before START_LCP_REQ is sent, so the guard was also satisfied when a participant failed while the master was still waiting for its START_LCP_CONF. The master then sent LCP_FRAG_ORD to the surviving LQHs from the node failure handling. The same node failure handling queues a START_LCP_CONF for the failed node, which completes the handshake and releases the start LCP mutex, and startLcpRoundLoopLab then failed ndbrequire(noOfStartedChkpt == 0) for the first live node, error 2341 in the master. This only happens on a master that has not yet run an LCP round, typically a master that has just taken over: a master that has completed a round still has all participants in m_allReplicasQueuedLQH, so the premature call sends nothing. Require c_lcp_runs_with_pause_support as well. It is set when the master has passed the START_LCP_REQ step, right before the LCP round starts, both in the normal LCP start and in LCP master takeover, and it is cleared when the LCP completes. Before the round there is nothing to drive: the round starts with the reduced set of participants once the handshake completes. Assert that the start LCP mutex has been released when the call is made. Reproduced with testNodeRestart -n LcpStartParticipantFail. How to repeat: See above Suggested fix: + * + * It must also not be called before the master has entered the LCP + * round. m_participatingLQH is set at the very start of the LCP, + * before START_LCP_REQ is sent, so a participant can fail while the + * master is still waiting for START_LCP_CONF. Starting fragment + * checkpoints here in that case makes startLcpRoundLoopLab, which is + * reached through the START_LCP_CONF queued above for the failed + * node and the asynchronous release of the start LCP mutex, fail + * ndbrequire(noOfStartedChkpt == 0). There is nothing to drive + * here in that case either: the round starts with the reduced set + * of participants once the handshake completes. + * + * c_lcp_runs_with_pause_support is set when the master has passed + * the START_LCP_REQ step, right before the LCP round starts, both + * in the normal LCP start and in LCP master takeover, and it is + * cleared when the LCP completes. */ - if (check_more_start_lcp && c_lcpMasterTakeOverState.state == LMTOS_IDLE) { + if (check_more_start_lcp && c_lcpMasterTakeOverState.state == LMTOS_IDLE && + c_lcp_runs_with_pause_support) { jam(); ndbrequire(isMaster()); + /* The start LCP mutex is released before the LCP round starts */ + ndbassert(c_startLcpMutexHandle.isNull()); startNextChkpt(signal); } } // Dbdih::execNODE_FAILREP()