Description:
RONDB-1111: Fix duplicate and lost TAKE_OVERTCCONF at TC take over master failure
The TC take over of a failed node's transactions is run by the master
node's DBTC. When a take over completes, the master broadcasts
TAKE_OVERTCCONF to all alive nodes, which remove the failed node from
their take over queues and complete the NF_TAKEOVER step of node
failure handling.
The broadcast is not atomic: if the master fails right after
completing a take over, the broadcast can reach a subset of the
surviving nodes. The new master decides from its own take over queue
whether to redo the take over, which leads to two defects:
1. Duplicate CONF: if the new master missed the broadcast it redoes
the take over and broadcasts a TAKE_OVERTCCONF of its own. A
survivor that received the original has already removed its queue
entry and fails the ndbrequire(tcFailNo < senderTcFailNo) in
execTAKE_OVERTCCONF, crashing a healthy node.
2. Lost CONF: if the new master received the broadcast but another
survivor did not, nobody ever takes the node over again. That
survivor waits forever for NF_TAKEOVER, node failure handling
never completes (blocking the failed node from rejoining) and GCP
completion is blocked through c_ongoing_take_over_cnt.
Receiver side fix (unconditional): a TAKE_OVERTCCONF for a node that
is not in the take over queue, received when we have seen every node
failure the sender had seen, is recognised as a duplicate and
dropped. This requires that the NF_TAKEOVER step is already completed
for the node: queue membership and the NF_TAKEOVER bit are maintained
together, so anything else remains a real protocol breakage caught by
an ndbrequire. A new per-host fail number (HostRecord::m_fail_no)
additionally drops a CONF from an older failure incident of a node
that has since restarted and failed again.
Master side fix: at master take over the new master re-announces
TAKE_OVERTCCONF for every node failure take over it has seen
complete, so that survivors that missed the original broadcast are
unblocked. Take overs not seen complete are still in the take over
queue and are redone as before. The re-announcement is only sent to
nodes running a version that drops duplicates (25.10.18 in the 25.10
series, 26.2.10 in the 26.02 series); older nodes are skipped and
keep the old behaviour.
New error inserts for testing: 8308 crashes the take over master
shortly after broadcasting TAKE_OVERTCCONF (delayed so the broadcast
is flushed first), 8309 discards one incoming TAKE_OVERTCCONF on a
chosen receiver. Together they simulate a master failing with a
partially delivered broadcast.
How to repeat:
See above
Suggested fix:
}
if (masterTakeOver &&
(instance() == 0 || instance() == TAKE_OVER_INSTANCE)) {
jam();
/**
* We have just become master. The old master may have completed
* take overs and broadcast TAKE_OVERTCCONF without the broadcast
* reaching every node before it failed. Re-announce every take
* over that we have seen complete so that no node is left waiting
* forever for a TAKE_OVERTCCONF that was lost with the old master
* (nodes that already received the original drop the duplicate).
* Take overs that we have not seen complete are still in our take
* over queue and are redone the normal way, which produces a
* TAKE_OVERTCCONF of its own.
*/
rebroadcast_take_overtcconf(signal);
}
if (tcFailNo >= senderTcFailNo) {
jam();
/**
* We have seen every node failure that the sender had seen
* when it completed the take over, so the missing queue
* entry cannot be explained by a NODE_FAILREP that we have
* not yet received. The consistent explanation is that
* this is a duplicate CONF: the previous master completed
* the take over and broadcast TAKE_OVERTCCONF, but failed
* before the broadcast reached all nodes. The new master
* then took the node over again, or re-announced the
* completed take over at master take over, and broadcast a
* CONF of its own to all alive nodes since it cannot know
* which nodes the original broadcast reached. We received
* the original, thus we have already handled the take over
* completion and can safely drop the duplicate. Anything
* else is a protocol breakage caught by the ndbrequire
* below, since queue membership and the NF_TAKEOVER bit are
* maintained together.
*/
ndbrequire((hostptr.p->m_nf_bits & HostRecord::NF_TAKEOVER) == 0);
g_eventLogger->info(
"DBTC %u: Dropping duplicate TAKE_OVERTCCONF for node %u"
" from 0x%x, sender fail no %u, own fail no %u",
instance(), hostptr.i, senderRef, senderTcFailNo, tcFailNo);
return;
}
© 2026, Oracle Corporation and/or its affiliates
Description: RONDB-1111: Fix duplicate and lost TAKE_OVERTCCONF at TC take over master failure The TC take over of a failed node's transactions is run by the master node's DBTC. When a take over completes, the master broadcasts TAKE_OVERTCCONF to all alive nodes, which remove the failed node from their take over queues and complete the NF_TAKEOVER step of node failure handling. The broadcast is not atomic: if the master fails right after completing a take over, the broadcast can reach a subset of the surviving nodes. The new master decides from its own take over queue whether to redo the take over, which leads to two defects: 1. Duplicate CONF: if the new master missed the broadcast it redoes the take over and broadcasts a TAKE_OVERTCCONF of its own. A survivor that received the original has already removed its queue entry and fails the ndbrequire(tcFailNo < senderTcFailNo) in execTAKE_OVERTCCONF, crashing a healthy node. 2. Lost CONF: if the new master received the broadcast but another survivor did not, nobody ever takes the node over again. That survivor waits forever for NF_TAKEOVER, node failure handling never completes (blocking the failed node from rejoining) and GCP completion is blocked through c_ongoing_take_over_cnt. Receiver side fix (unconditional): a TAKE_OVERTCCONF for a node that is not in the take over queue, received when we have seen every node failure the sender had seen, is recognised as a duplicate and dropped. This requires that the NF_TAKEOVER step is already completed for the node: queue membership and the NF_TAKEOVER bit are maintained together, so anything else remains a real protocol breakage caught by an ndbrequire. A new per-host fail number (HostRecord::m_fail_no) additionally drops a CONF from an older failure incident of a node that has since restarted and failed again. Master side fix: at master take over the new master re-announces TAKE_OVERTCCONF for every node failure take over it has seen complete, so that survivors that missed the original broadcast are unblocked. Take overs not seen complete are still in the take over queue and are redone as before. The re-announcement is only sent to nodes running a version that drops duplicates (25.10.18 in the 25.10 series, 26.2.10 in the 26.02 series); older nodes are skipped and keep the old behaviour. New error inserts for testing: 8308 crashes the take over master shortly after broadcasting TAKE_OVERTCCONF (delayed so the broadcast is flushed first), 8309 discards one incoming TAKE_OVERTCCONF on a chosen receiver. Together they simulate a master failing with a partially delivered broadcast. How to repeat: See above Suggested fix: } if (masterTakeOver && (instance() == 0 || instance() == TAKE_OVER_INSTANCE)) { jam(); /** * We have just become master. The old master may have completed * take overs and broadcast TAKE_OVERTCCONF without the broadcast * reaching every node before it failed. Re-announce every take * over that we have seen complete so that no node is left waiting * forever for a TAKE_OVERTCCONF that was lost with the old master * (nodes that already received the original drop the duplicate). * Take overs that we have not seen complete are still in our take * over queue and are redone the normal way, which produces a * TAKE_OVERTCCONF of its own. */ rebroadcast_take_overtcconf(signal); } if (tcFailNo >= senderTcFailNo) { jam(); /** * We have seen every node failure that the sender had seen * when it completed the take over, so the missing queue * entry cannot be explained by a NODE_FAILREP that we have * not yet received. The consistent explanation is that * this is a duplicate CONF: the previous master completed * the take over and broadcast TAKE_OVERTCCONF, but failed * before the broadcast reached all nodes. The new master * then took the node over again, or re-announced the * completed take over at master take over, and broadcast a * CONF of its own to all alive nodes since it cannot know * which nodes the original broadcast reached. We received * the original, thus we have already handled the take over * completion and can safely drop the duplicate. Anything * else is a protocol breakage caught by the ndbrequire * below, since queue membership and the NF_TAKEOVER bit are * maintained together. */ ndbrequire((hostptr.p->m_nf_bits & HostRecord::NF_TAKEOVER) == 0); g_eventLogger->info( "DBTC %u: Dropping duplicate TAKE_OVERTCCONF for node %u" " from 0x%x, sender fail no %u, own fail no %u", instance(), hostptr.i, senderRef, senderTcFailNo, tcFailNo); return; }