Description:
Fix permanent node id lockout after a node fails during its start
When a data node fails while it is still starting (its DIH nodeStatus
is STARTING or DEAD, not ALIVE), Dbdih::failedNodeSynchHandling takes
the "node not even started" branch: the node record is marked DEAD
since the node was never part of the node failure protocols. However,
the m_NF_COMPLETE_REP signal counter has already been armed at that
point with every then-alive node as a waiter, and the branch does not
clear it.
Normally this is harmless: all waiters are alive, each of them
completes its local failure handling of the failed node and broadcasts
NF_COMPLETEREP(blockNo=0), which drains the counter, and
nodeFailCompletedCheckLab reports NDB_FAILCONF to QMGR.
But if one of the waiters itself fails before sending its broadcast,
its bit can never be cleared: the waiter's next incarnation is a new
entity that owes no reports, and the dead-waiter rescue in
failedNodeSynchHandling only considered waiting nodes in state DYING,
skipping records in state DEAD. The counter then never drains, DBDIH
never sends NDB_FAILCONF, QMGR's failState for the failed node stays
WAITING_FOR_NDB_FAILCONF, and every attempt to re-allocate the node's
node id is refused with error 1703 - forever. The node cannot rejoin
until the whole cluster is restarted. The cluster log shows the
signature warning:
Failure handling of node N has not completed in X seconds - state = 6
This interleaving occurs in practice during rolling restarts: one
node's replacement dies while climbing through its start phases (for
example with error 2308 or 2341 caused by another node's failure), and
a node that owed an NF_COMPLETEREP for that death is taken down by the
same rolling restart within the following second.
Fix: allow the dead-waiter rescue to cover waiting nodes in state DEAD
as well as DYING. This is safe because the counter is populated only
in failedNodeSynchHandling (always after a full clear) and drained
only in execNF_COMPLETEREP, so a DEAD node with a non-empty counter
can only be a node that failed before completing its start; a normally
completed DEAD node has an empty counter and isWaitingFor() is false.
The change also heals an already leaked counter: the rescue runs
whenever any node fails, so the next failure of the stuck waiter's
node id drains the leaked bit through the standard path.
Verified with the NodeFailLeakDuringNodeStart test case added to
testNodeRestart: without the fix the test wedges the failed node's
node id permanently (error 1703); with the fix the node rejoins
within seconds.
How to repeat:
See above
Suggested fix:
failedNodePtr.p->m_NF_COMPLETE_REP.setWaitingFor(nodePtr.i);
} else {
jam();
- if ((nodePtr.p->nodeStatus == NodeRecord::DYING) &&
+ if ((nodePtr.p->nodeStatus == NodeRecord::DYING ||
+ nodePtr.p->nodeStatus == NodeRecord::DEAD) &&
(nodePtr.p->m_NF_COMPLETE_REP.isWaitingFor(failedNodePtr.i))) {
jam();
/*----------------------------------------------------*/
@@ -9791,6 +9792,9 @@ void Dbdih::failedNodeSynchHandling(Signal *signal,
/* REPORT THAT NODE FAILURE HANDLING WAS */
/* COMPLETED ON THE NEW FAILED NODE FOR THIS */
/* PARTICULAR OLD FAILED NODE. */
/* A DEAD NODE CAN ALSO BE WAITING: A NODE THAT */
/* FAILED WHILE STARTING IS MARKED DEAD BELOW */
/* WITH ITS MASK STILL ARMED. */
/*----------------------------------------------------*/
NFCompleteRep *const nf = (NFCompleteRep *)&signal->theData[0];
nf->blockNo = 0;
© 2026, Oracle Corporation and/or its affiliates
Description: Fix permanent node id lockout after a node fails during its start When a data node fails while it is still starting (its DIH nodeStatus is STARTING or DEAD, not ALIVE), Dbdih::failedNodeSynchHandling takes the "node not even started" branch: the node record is marked DEAD since the node was never part of the node failure protocols. However, the m_NF_COMPLETE_REP signal counter has already been armed at that point with every then-alive node as a waiter, and the branch does not clear it. Normally this is harmless: all waiters are alive, each of them completes its local failure handling of the failed node and broadcasts NF_COMPLETEREP(blockNo=0), which drains the counter, and nodeFailCompletedCheckLab reports NDB_FAILCONF to QMGR. But if one of the waiters itself fails before sending its broadcast, its bit can never be cleared: the waiter's next incarnation is a new entity that owes no reports, and the dead-waiter rescue in failedNodeSynchHandling only considered waiting nodes in state DYING, skipping records in state DEAD. The counter then never drains, DBDIH never sends NDB_FAILCONF, QMGR's failState for the failed node stays WAITING_FOR_NDB_FAILCONF, and every attempt to re-allocate the node's node id is refused with error 1703 - forever. The node cannot rejoin until the whole cluster is restarted. The cluster log shows the signature warning: Failure handling of node N has not completed in X seconds - state = 6 This interleaving occurs in practice during rolling restarts: one node's replacement dies while climbing through its start phases (for example with error 2308 or 2341 caused by another node's failure), and a node that owed an NF_COMPLETEREP for that death is taken down by the same rolling restart within the following second. Fix: allow the dead-waiter rescue to cover waiting nodes in state DEAD as well as DYING. This is safe because the counter is populated only in failedNodeSynchHandling (always after a full clear) and drained only in execNF_COMPLETEREP, so a DEAD node with a non-empty counter can only be a node that failed before completing its start; a normally completed DEAD node has an empty counter and isWaitingFor() is false. The change also heals an already leaked counter: the rescue runs whenever any node fails, so the next failure of the stuck waiter's node id drains the leaked bit through the standard path. Verified with the NodeFailLeakDuringNodeStart test case added to testNodeRestart: without the fix the test wedges the failed node's node id permanently (error 1703); with the fix the node rejoins within seconds. How to repeat: See above Suggested fix: failedNodePtr.p->m_NF_COMPLETE_REP.setWaitingFor(nodePtr.i); } else { jam(); - if ((nodePtr.p->nodeStatus == NodeRecord::DYING) && + if ((nodePtr.p->nodeStatus == NodeRecord::DYING || + nodePtr.p->nodeStatus == NodeRecord::DEAD) && (nodePtr.p->m_NF_COMPLETE_REP.isWaitingFor(failedNodePtr.i))) { jam(); /*----------------------------------------------------*/ @@ -9791,6 +9792,9 @@ void Dbdih::failedNodeSynchHandling(Signal *signal, /* REPORT THAT NODE FAILURE HANDLING WAS */ /* COMPLETED ON THE NEW FAILED NODE FOR THIS */ /* PARTICULAR OLD FAILED NODE. */ /* A DEAD NODE CAN ALSO BE WAITING: A NODE THAT */ /* FAILED WHILE STARTING IS MARKED DEAD BELOW */ /* WITH ITS MASK STILL ARMED. */ /*----------------------------------------------------*/ NFCompleteRep *const nf = (NFCompleteRep *)&signal->theData[0]; nf->blockNo = 0;