Bug #121185 Crash after master node failure
Submitted: 28 Aug 10:27
Reporter: Mikael Ronström Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:8.4.11 OS:Any
Assigned to: CPU Architecture:Any

[28 Aug 10:27] Mikael Ronström
Description:
    RONDB-1114: Do not start LCP fragment checkpoints before the LCP round
    
    When an LCP participant fails, execNODE_FAILREP calls startNextChkpt
    so that the LCP does not hang waiting for the failed node. The guard
    only checked that no LCP master takeover was in progress.
    
    m_participatingLQH is set at the very start of the LCP, before
    START_LCP_REQ is sent, so the guard was also satisfied when a
    participant failed while the master was still waiting for its
    START_LCP_CONF. The master then sent LCP_FRAG_ORD to the surviving
    LQHs from the node failure handling. The same node failure handling
    queues a START_LCP_CONF for the failed node, which completes the
    handshake and releases the start LCP mutex, and startLcpRoundLoopLab
    then failed ndbrequire(noOfStartedChkpt == 0) for the first live
    node, error 2341 in the master.
    
    This only happens on a master that has not yet run an LCP round,
    typically a master that has just taken over: a master that has
    completed a round still has all participants in
    m_allReplicasQueuedLQH, so the premature call sends nothing.
    
    Require c_lcp_runs_with_pause_support as well. It is set when the
    master has passed the START_LCP_REQ step, right before the LCP round
    starts, both in the normal LCP start and in LCP master takeover, and
    it is cleared when the LCP completes. Before the round there is
    nothing to drive: the round starts with the reduced set of
    participants once the handshake completes. Assert that the start LCP
    mutex has been released when the call is made.
    
    Reproduced with testNodeRestart -n LcpStartParticipantFail.
    

How to repeat:
See above

Suggested fix:
+   *
+   * It must also not be called before the master has entered the LCP
+   * round. m_participatingLQH is set at the very start of the LCP,
+   * before START_LCP_REQ is sent, so a participant can fail while the
+   * master is still waiting for START_LCP_CONF. Starting fragment
+   * checkpoints here in that case makes startLcpRoundLoopLab, which is
+   * reached through the START_LCP_CONF queued above for the failed
+   * node and the asynchronous release of the start LCP mutex, fail
+   * ndbrequire(noOfStartedChkpt == 0). There is nothing to drive
+   * here in that case either: the round starts with the reduced set
+   * of participants once the handshake completes.
+   *
+   * c_lcp_runs_with_pause_support is set when the master has passed
+   * the START_LCP_REQ step, right before the LCP round starts, both
+   * in the normal LCP start and in LCP master takeover, and it is
+   * cleared when the LCP completes.
    */
-  if (check_more_start_lcp && c_lcpMasterTakeOverState.state == LMTOS_IDLE) {
+  if (check_more_start_lcp && c_lcpMasterTakeOverState.state == LMTOS_IDLE &&
+      c_lcp_runs_with_pause_support) {
     jam();
     ndbrequire(isMaster());
+    /* The start LCP mutex is released before the LCP round starts */
+    ndbassert(c_startLcpMutexHandle.isNull());
     startNextChkpt(signal);
   }
 }  // Dbdih::execNODE_FAILREP()