Bug #121176 Copy scan hangs

• Submitted: 26 Aug 15:34
• Modified: 27 Aug 12:24
• Reporter: Mikael Ronström
• Status: Open
• Category: MySQL Cluster: Cluster (NDB) storage engine
• Severity: S3 (Non-critical)
• Version: 8.4.11
• OS: Any
• CPU Architecture: Any

[26 Aug 15:34] Mikael Ronström

Description:
When the copy target fails while a fragment copy is waiting for a local scan response, closeCopyRequestLab marks the scan completed and clears the outstanding word credits.

nextScanConfCopyLab checked the completion marker only in one no-row response path. A fetched row could therefore still be processed and sent to the failed target. This added new copyCountWords after failure handling had run, leaving closeCopyLab waiting for credits that could never arrive.

Check the completion marker before processing any scan response. Discard the returned record, clear credits from any send racing with the close request, and close the copy scan immediately.

Observed with ndb.ndb_TCtakeover_stall.

How to repeat:
See above

Suggested fix:
void Dblqh::nextScanConfCopyLab(Signal *signal,
                                const TcConnectionrecPtr tcConnectptr) {
  NextScanConf *const nextScanConf = (NextScanConf *)&signal->theData[0];
+  if (unlikely(scanptr.p->scanCompletedStatus == ZTRUE)) {
+    jam();
+    /*
+     * The copy target failed while a local fetch was outstanding.
+     * Do not process the fetched record or send it to the failed node.
+     * Discard credits from any send that raced with the close request.
+     */
+    tcConnectptr.p->copyCountWords = 0;
+    closeCopyLab(signal, tcConnectptr.p);
+    return;
+  }
  if (nextScanConf->fragId == RNIL) {

[27 Aug 12:24] Mikael Ronström

Here is one more patch needed to fix it (require some modification to fit in NDB:

    RONDB-1112: Refuse COPY_FRAGREQ for a failed target node
    
    Completes the fix for the copy scan stall after the copy target fails.
    The previous fix guarded nextScanConfCopyLab() on scanCompletedStatus,
    but a COPY_FRAGREQ racing the node failure re-initializes the copy scan
    record — resetting scanCompletedStatus — and thereby wipes a close
    already requested by closeCopyRequestLab() for the previous fragment.
    The freshly initialized copy then streams operations to the dead node,
    re-inflating copyCountWords with flow-control credits that can never be
    returned (the credit is echoed in the peer's LQHKEYCONF), and the scan
    parks permanently in WAIT_LQHKEY_COPY: node failure handling has
    already completed, so nothing ever closes it. Observed with
    ndb.ndb_TCtakeover_stall: with the previous fix active, 75 copy sends
    to the dead node within seconds of the take-over close, then a
    permanent stall until test timeout.
    
    Fix, two layers:
    - execCOPY_FRAGREQ(): if the copy target node is already known to have
      failed (get_node_status() != ZNODE_UP), reply COPY_FRAGREF with
      ZNODE_FAILURE_ERROR before touching the copy scan record, so the
      racing request can neither wipe a pending close nor start a copy to
      a dead node. The master's copy process handles the REF as it handles
      any failed copy target.
    - nextScanConfCopyLab(): the close guard also checks
      get_node_status(scanNodeId), catching a copy whose target died
      without scanCompletedStatus being (or staying) set.
    
    With both layers in place a full test suite run passed with no copy
    stalls; previously the same suite reproduced the stall reliably.
    Independent of any RonDB fiber configuration; applies to upstream
    branches.
    
    Co-Authored-By: Claude Fable 5 <[e-mail]>

diff --git a/storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp b/storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
index a7270cdb1b4..47083dc592f 100644
--- a/storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
+++ b/storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
@@ -21509,6 +21509,46 @@ void Dblqh::execCOPY_FRAGREQ(Signal *signal) {
   const Uint32 maxPage = copyFragReq->nodeList[nodeCount];
   const Uint32 requestInfo = copyFragReq->nodeList[nodeCount + 1];
 
+  if (unlikely(get_node_status(nodeId) != ZNODE_UP)) {
+    jam();
+    /**
+     * The copy target is already known to have failed. Refuse before
+     * touching the copy scan record: a COPY_FRAGREQ racing the node
+     * failure would otherwise re-initialize the scan record — wiping
+     * the scanCompletedStatus set by closeCopyRequestLab() for the
+     * previous fragment's close — and then stream copy operations to
+     * the dead node, re-inflating copyCountWords with credits that can
+     * never return and parking the scan in WAIT_LQHKEY_COPY forever
+     * (observed in ndb.ndb_TCtakeover_stall: 75 sends within seconds
+     * after the takeover close, then a permanent stall). The failure
+     * handling that is already underway cleans up everything else; the
+     * master's copy process handles the REF as it handles any failed
+     * copy target.
+     */
+    CopyFragRef *const ref = (CopyFragRef *)&signal->theData[0];
+    ref->userPtr = copyPtr;
+    ref->sendingNodeId = cownNodeid;
+    ref->startingNodeId = nodeId;
+    ref->tableId = tabptr.i;
+    ref->fragId = fragId;
+    ref->errorCode = ZNODE_FAILURE_ERROR;
+    sendSignal(userRef, GSN_COPY_FRAGREF, signal, CopyFragRef::SignalLength,
+               JBB);
+
+    /*
+     * start_new_copyFragReq() reserves an active slot before sending a
+     * queued request to this block. Release that reservation when the
+     * request is rejected before a copy scan is created.
+     */
+    if (from_queue)
+    {
+      ndbrequire(c_active_copyFragReq > 0);
+      adjust_copyFragReq_rates(false);
+      start_new_copyFragReq(signal);
+    }
+    return;
+  }
+
   if (requestInfo == CopyFragReq::CFR_NON_TRANSACTIONAL) {
     jam();
   } else {
@@ -21932,12 +21972,20 @@ void Dblqh::accScanConfCopyLab(Signal *signal) {
 void Dblqh::nextScanConfCopyLab(Signal *signal,
                                 const TcConnectionrecPtr tcConnectptr) {
   NextScanConf *const nextScanConf = (NextScanConf *)&signal->theData[0];
-  if (unlikely(scanptr.p->scanCompletedStatus == ZTRUE)) {
+  if (unlikely(scanptr.p->scanCompletedStatus == ZTRUE ||
+               get_node_status(scanptr.p->scanNodeId) != ZNODE_UP)) {
     jam();
     /*
      * The copy target failed while a local fetch was outstanding.
      * Do not process the fetched record or send it to the failed node.
      * Discard credits from any send that raced with the close request.
+     *
+     * The node-status check additionally catches a copy whose target
+     * died without scanCompletedStatus being (or staying) set: a
+     * COPY_FRAGREQ racing the node failure re-initializes the scan
+     * record, wiping the flag an earlier closeCopyRequestLab() set —
+     * observed as 75 copy sends to the dead node within seconds of the
+     * take-over close, ending in a permanent WAIT_LQHKEY_COPY stall.
      */
     tcConnectptr.p->copyCountWords = 0;
     closeCopyLab(signal, tcConnectptr.p);