Description:
1. SUMMARY
An ndbmtd data node (NDB 8.4.11, release build) aborted three times with
DbtupRoutines.cpp:666: require((srcBytes <= max_var_size)) failed
- 3 events, 2 data nodes: node 4 twice (2026-08-31, 2026-09-03), node 2 once (2026-10-17);
different node groups, separate hosts, RAM from different manufacturers.
- Same failure site: varsize_reader at base+0x39d8d1; ndbmtd md5 identical on both hosts.
- Always a query thread (DBQLQH) doing a READ COMMITTED read of a var-size attribute: two
ordered-index scans and one primary-key dirty read, 2 tables, 3 fragments.
- In each trace dump the LDM thread owning that fragment was in the middle of an insert into
the same fragment; its JAM shows it had just run Dbtup::alloc_var_part on an existing var
page (cause or consequence, see caveat (b)).
- The partner replica kept the node group up; with StopOnError=0 the node was back in 69 s and
84 s (event 1: ~2 h, StopOnError was still 1). We found no loss of stored data.
Hypothesis (source reading, unproven): insert-path var-page reorganisation (alloc_var_part ->
Tup_varsize_page::alloc_record -> reorg) runs without exclusive fragment access, while
query-thread readers dereference var-part pointers after releasing the TUP fragment mutex.
See "Suggested fix" for file:line references.
Attachment ndb-varsize-assert-bundle.tgz (README.txt inside): per event the error log, an
out.log excerpt, the crashing query-thread and owning-LDM traces; config.ini; full table
definitions; and the full-length versions of this description and of the suggested fix
(full-description.txt, full-suggested-fix.txt). Everything is anonymised.
2. ENVIRONMENT
mysql-cluster-community 8.4.11-1ubuntu24.04 (repo.mysql.com apt), Ubuntu 24.04, kernel 6.8.0.
Hosts: single-socket AMD EPYC 4004-series (Zen 4), 16C/32T, 2 x 32 GB DDR5 (no ECC beyond
DDR5 on-die ECC), NVMe. 6 data nodes, NoOfReplicas=2, NG0={1,2} NG1={3,4} NG2={5,6};
3 management nodes; one SQL node per data node host (ndb_cluster_connection_pool=4).
AutomaticThreadConfig=1, the same on all 6 data nodes: NumCPUs=16 until a rolling restart
on 2026-09-03 (events 1-2: 4 ldm, 4 query = DBQLQH 1-4), NumCPUs=24 since then (event 3:
6 ldm thr 2-7 = DBLQH 1-6, 6 query thr 8-13 = DBQLQH 1-6).
DataMemory=32G (40G at event 1), TransactionMemory=3G, LockPagesInMainMemory=1,
StopOnError=0 (1 at event 1), MaxNoOfConcurrentOperations=262144, MaxNoOfConcurrentScans=500.
SQL nodes: READ-COMMITTED, ndb_batch_size=1M, ndb_read_backup=ON.
3. THE THREE EVENTS ("35/3" = table 35 fragment 3)
Event 1 Event 2 Event 3
Time (UTC) 2026-08-31 13:58:44 2026-09-03 21:11:23 2026-10-17 23:24:36
Node / thread 4 / thr 9 DBQLQH(4) 4 / thr 8 DBQLQH(3) 2 / thr 8 DBQLQH(1)
Read ordered-index scan PK read: LQHKEYREQ ordered-index range
DESC, no bound, Read Simple Dirty scan ASC; died on 1st
pushed condition ACC_CHECK_SCAN after
its SCAN_FRAGREQ
Object index 142 (idx_ts), table 107, 107/4 index 141 (PRIMARY),
35/10 35/3
Requester API 154 via DBTC(2) API 160 via DBTC(1) API 177 via DBTC(2)
on node 2 on node 4 (local) on node 5
Abort path interpreterNextLab > handleReadReq > interpreterStartLab >
readAttributes > readAttributes > readAttributes >
varsize_reader read_packed > read_packed >
varsize_reader varsize_reader
Owning LDM thr 5 = DBLQH(4): thr 3 = DBLQH(2): thr 7 = DBLQH(6):
Write 35/10 -> ACC Insert 107/4: Write 35/3 -> ACC
insert; alloc_fix_rec alloc_fix_rec + insert; alloc_fix_rec
+ alloc_var_part alloc_var_part + alloc_var_part
(existing page); then (existing page); then (existing page); then
waits at DblqhMain COMMIT waits at waits at DblqhMain
7105 (write-key) DblqhMain 7209 7105 (write-key)
Caveats: (a) the crashing node always held the primary replica; with read_backup=1 that is
just DBTC read routing here, not a clue. (b) A stuck owning LDM is also the expected
consequence of any query-thread crash (the dead thread never releases fragment access), so
it fits a race but does not prove one.
4. OUT.LOG (full stacks and base-relative offsets of all three events in the attachment)
Event 3 stack: varsize_reader < read_packed < readAttributes < interpreterStartLab <
execTUPKEYREQ < Dblqh::nextScanConfScanLab < continue_next_scan_conf < Dbtux::continue_scan
< Dblqh::execACC_CHECK_SCAN. All three error logs: Error 6000, "Signal 6 received; Aborted".
5. TABLES (anonymised; full DDL of both tables in the attachment)
Table id 35 (t1; indexes 141 = PRIMARY, 142 = idx_ts, 86 = idx_k1_ts), 13 columns:
PRIMARY KEY (k1 varchar(12) latin1, k2 char(32) latin1), PARTITION BY KEY (k1), utf8mb4
default; var-size: k1, s1 + s2 varchar(255) COLUMN_FORMAT DYNAMIC, s3 varchar(45),
s4 varchar(32), s5 varchar(20) (static); fixed: ts int (idx_ts, idx_k1_ts = (k1,ts)) and
small int/decimal/char(2) columns. NDB_TABLE=NOLOGGING=1.
Table id 107 (t2; index 114 = PRIMARY): same PK and partitioning, 28 columns; varchar
s1..s14, of which s1, s2, s4 (varchar(255)) and s9 (varchar(64)) are DYNAMIC, declared at
CREATE TABLE on 2026-08-29.
Both tables: NOLOGGING, Row Checksum 1, ForceVarPart 1, FragmentCount 12, FOR_RP_BY_LDM,
read_backup 1, no BLOB/TEXT/JSON; each mixes static var-size and DYNAMIC columns.
6. WORKLOAD (high-churn session-style tables)
t1: several hundred thousand rows. All 6 SQL nodes write it with multi-row REPLACE (~1000 rows
per statement, on the order of 100k rows/min); a REPLACE of a missing key becomes an insert.
Rows expire a few minutes after their last write; once a minute one SQL node deletes them in
chunks (READ COMMITTED ordered read on idx_ts, then a range DELETE on ts). Readers are
READ COMMITTED per-k1 queries (e.g. "WHERE k1 = ? ORDER BY ts DESC LIMIT n"), which DBTC can
route to query threads. t2 (tens of thousands of rows): single-row PK reads and writes. The
statements behind the crashing reads were not identified.
7. RULED OUT
- OOM / memory pressure, watchdog / overload / GCP warnings: none before any abort.
- Index-statistics scans: stat 0, API requesters, and none running at event 3.
- One table / index / node: 2 tables, 3 access paths (141 ASC, 142 DESC, hash PK), 2 nodes.
- Hardware (unlikely, not excluded): no MCE/EDAC/AER events, HardwareCorrupted 0 kB on all 6
hosts; different DIMM makers on nodes 2 and 4; node 4 quiet for 44 days after 2 events in
3 days; Row Checksum covers the fixed part incl. the var part reference. But the RAM is
non-ECC, so a bit flip in any one event cannot be strictly excluded.
- Changes: no package or config change on the day of event 3.
8. IMPACT
The node was rebuilt from its partner; SQL clients stalled ~6 s until the node was declared
dead, and in-flight transactions got temporary errors (4010, 4028, 4119, 270). If the race
hypothesis is right, some reads could also return wrong var-size values with no error
instead of tripping the require; we have no evidence either way.
How to repeat:
We cannot reproduce this on demand; it is not deterministic. Below are the conditions under
which it happened, and a stress setup that should make the suspected race more likely.
Observed rate: 3 events in about 350 data-node-days (6 data nodes, about two months of
production): 2026-08-31, 2026-09-03 and 2026-10-17. The events hit two different nodes. The
thread layout was NumCPUs=16 for the first two and NumCPUs=24 for the third.
Conditions in production (DDL and settings are in the Description):
1. ndbmtd with query threads: AutomaticThreadConfig=1, one query thread per LDM thread,
NoOfReplicas=2, read_backup=1, 12 fragments per table (FOR_RP_BY_LDM).
2. NOLOGGING tables with a 2-column PK (k1 varchar(12), k2 char(32)), PARTITION BY KEY (k1),
extra ordered indexes (t1: on (ts) and, from after event 1, on (k1, ts); t2: on (n3, ts)),
and several var-size columns: static varchar columns plus COLUMN_FORMAT DYNAMIC
varchar(255) columns whose values vary in length.
3. Writers (all 6 SQL nodes): continuous multi-row REPLACE of about 1000 rows per statement,
on the order of 100k rows/min. A REPLACE of a key that is not present becomes an INSERT,
which allocates a var part (alloc_var_part). On the second table: single-row writes,
e.g. the INSERT of a new key in event 2. Its expiry job also runs an UPDATE (by PK list)
that sets a DYNAMIC varchar column to ''.
4. Expiry: every minute, rows older than a few minutes are deleted in chunks of 10000 rows
(an ordered READ COMMITTED snapshot via the (ts) index, then a range DELETE on ts). As a
result, var pages are constantly freed and re-filled, so free space inside them is
fragmented.
5. Readers: many concurrent READ COMMITTED statements, which DBTC can route to query
threads, reading var-size columns of the same fragments:
a. primary-key lookups (WHERE k1 = ? AND k2 = ?): the access type of event 2
b. single-partition range scans on PRIMARY (WHERE k1 = ?): the access type of event 3
c. ordered scans with a pushed condition on a var-size column, e.g.
WHERE k1 = ? ORDER BY ts DESC LIMIT n through the (ts) index: the access type of
event 1
The actual statements were not identified.
Suggested stress setup (we cannot run debug builds in production):
- Concentrate the race on few fragments: 1-2 LDM threads with the maximum number of query
threads (query = 2x or 3x ldm). Use a single-fragment table, or make all readers and
writers hit one partition key range.
- Make Tup_varsize_page::reorg() frequent. Insert rows whose var parts vary strongly in size
(0-255 bytes in several columns), delete a random half, and keep inserting. That way
alloc_record() often finds enough total free space but not enough contiguous space
(alloc_size >= largest_size).
- Run 16-64 client threads doing READ COMMITTED PK reads and k1-range scans that return all
var-size columns, concurrently with 4-8 threads doing INSERT/REPLACE of new keys plus
DELETEs.
- Make silent corruption detectable: store each var column as a deterministic function of
(k1, k2), e.g. a repeated hash of length f(k1,k2). Every reader then checks what it
receives. If the hypothesis is right, mismatches may also appear, possibly before the
require fires.
- In a debug build: add a jam()/counter to Tup_varsize_page::reorg(), enable the #if 0
consistency check in varsize_reader (DbtupRoutines.cpp:673-692), and if possible delay
query-thread reads between DbtupExecQuery.cpp:1209 (mutex released) and readAttributes()
to widen the window.
Expected result if the hypothesis holds: DbtupRoutines.cpp:666 require((srcBytes <=
max_var_size)) in a DBQLQH thread, and/or reads returning var-column values that belong to
other rows.
Suggested fix:
HYPOTHESIS FROM SOURCE READING, NOT A VERIFIED ROOT CAUSE. (Longer version with quoted code:
full-suggested-fix.txt in the attachment.)
References are to tag mysql-cluster-8.4.11, storage/ndb/src/kernel/blocks/dbtup/ unless
another path is given. We have not reproduced the crash; a release build cannot show whether
a page reorg happened (tuppage.cpp has no jam()). By our diff, DbtupRoutines.cpp,
DbtupExecQuery.cpp, DbtupVarAlloc.cpp, DbtupCommit.cpp, Dbtup.hpp, Dblqh.hpp and DblqhMain.cpp
are unchanged in mysql-cluster-26.7.0 and the 9.7 branch.
A. The failing check
DbtupRoutines.cpp:666 require(srcBytes <= max_var_size) in varsize_reader. srcBytes is the
difference of two Uint16 entries of the row's offset array, read directly from the var page
(readVarSizeNotNULL 791-795; readDynVarSizeShrunkenNotNULL 1319-1321). If those bytes no
longer hold this row's offset array, the difference can exceed the column maximum (trips the
require) or come out small and plausible (silently returns another row's bytes).
B. Query-thread readers do not hold the mutex while they read
DbtupExecQuery.cpp execTUPKEYREQ, ZREAD: 1183 acquire_frag_mutex_read; 1192 setup_read ->
prepare_read stores pointers INTO the var page (4458, 4491-4492: m_data_ptr,
m_offset_array_ptr); 1199-1208 checksum under the mutex; 1209 release_frag_mutex_read;
1211 handleReadReq -> readAttributes (or interpreterStartLab). Dbtup.hpp:856-867: in query
threads the mutex is tup_frag_mutex[logicalPageId & 3], hashed by the FIXED-part page.
4850-4855 states the assumption that holding the mutex only during prepare_read is enough.
C. The insert path can move other rows' var parts without exclusive fragment access
handleInsertReq (DbtupExecQuery.cpp:2458-2461) -> alloc_var_rec (DbtupVarAlloc.cpp:83-115)
-> alloc_fix_rec (takes only the mutex of the NEW row's fixed page, DbtupFixAlloc.cpp:128)
-> alloc_var_part (117-157): get_alloc_page returns an existing var page (jam 138, present in
all three LDM JAMs), then Var_page::alloc_record (147-149). tuppage.cpp:239-251
alloc_record: if (alloc_size >= largest_size) reorg(temp); reorg (368-396) re-packs every
live entry on the page, so the var parts of OTHER rows can move. No
upgrade_to_exclusive_frag_access() is taken on this path ("An initial INSERT operation
requires no mutex", DbtupExecQuery.cpp:1349-1355); fragment access is only upgraded later
(1362-1373, write-key, before the TUX insert), which is where the LDM was blocked in events
1 and 3 (DblqhMain.cpp:7103-7105). alloc_var_rowid (DbtupVarAlloc.cpp:563-590) is similar.
D. Contrast: other paths that move var data do take exclusive access
handle_size_change_after_update (DbtupExecQuery.cpp:4796-4857): its comment says var-part
reallocation "is intruding on all readers from the query thread ... it can even reorganise
an entire page", and it calls upgrade_to_exclusive_frag_access (4825) around realloc_var_part
(which can reorg or call alloc_var_part). Commit and dealloc are exclusive as well
(DbtupCommit.cpp:1782, 201). The insert path in C reaches alloc_var_part -> alloc_record ->
reorg without that protection.
E. move_var_part (OPTIMIZE TABLE path) appears to have the same gap
DbtupVarAlloc.cpp:365-367 calls alloc_record on the destination page (may reorg) before the
upgrade at 393, although its own comment (382-392) says all places that reorganise pages and
rows must use exclusive fragment access because query threads may be reading.
F. Fit with the events, and what does not prove it
- All three are query-thread reads through readAttributes -> varsize_reader on fragments
where the owning LDM's JAM shows alloc_var_part on an existing var page.
- A write prepare sets m_cond_write_key_waiters (Dblqh.hpp:5030) until write-key access is
granted (DblqhMain.cpp:7118), and a resuming scan cannot re-acquire access while it is set
(Dblqh.hpp:4867-4874). So in events 1 and 3 the blocked insert started after the scan last
resumed: during the failing row read (the race) or after the require fired.
- Not discriminating: allocations into existing var pages are common (18 in the event-3 LDM
JAMs), and a stuck owning LDM is also the expected consequence of any query-thread crash.
G. Possible fixes (suggestions only)
1. In alloc_var_part / alloc_record: when the allocation would reorg(), pick another or a
fresh var page, or take exclusive fragment access first as
handle_size_change_after_update does. alloc_var_rec calls alloc_var_part while holding the
mutex from alloc_fix_rec, and DbtupExecQuery.cpp:1368-1371 warns against upgrading while
holding a fragment mutex, so the decision probably has to be made before alloc_fix_rec, or
the page choice has to avoid reorg() altogether.
2. In move_var_part: upgrade_to_exclusive_frag_access before alloc_record on the destination.
3. Diagnostics: a jam() in Tup_varsize_page::reorg, and the disabled #if 0 consistency check
in varsize_reader (DbtupRoutines.cpp:673-692).
H. Questions
1. Is this known internally? We found no public bug. Related but already fixed in our version:
Bug #111932 / #35660890 (query-thread multi-range scan lost its fragment lock, 8.4.4).
2. If our reading is wrong, what keeps query threads off a fragment while its LDM runs
alloc_var_part?
3. Is disabling query threads (AutomaticThreadConfig=0, ClassicFragmentation=false, explicit
ThreadConfig with ldm={count=6}, query={count=0}) a supported and effective workaround?
4. Is a rolling restart that temporarily mixes nodes with and without query threads safe in
8.4.11 (remote DBTCs address DBQLQH instances directly)?
5. Should OPTIMIZE TABLE on tables with var-size columns be avoided until a fix (section E)?
Description: 1. SUMMARY An ndbmtd data node (NDB 8.4.11, release build) aborted three times with DbtupRoutines.cpp:666: require((srcBytes <= max_var_size)) failed - 3 events, 2 data nodes: node 4 twice (2026-08-31, 2026-09-03), node 2 once (2026-10-17); different node groups, separate hosts, RAM from different manufacturers. - Same failure site: varsize_reader at base+0x39d8d1; ndbmtd md5 identical on both hosts. - Always a query thread (DBQLQH) doing a READ COMMITTED read of a var-size attribute: two ordered-index scans and one primary-key dirty read, 2 tables, 3 fragments. - In each trace dump the LDM thread owning that fragment was in the middle of an insert into the same fragment; its JAM shows it had just run Dbtup::alloc_var_part on an existing var page (cause or consequence, see caveat (b)). - The partner replica kept the node group up; with StopOnError=0 the node was back in 69 s and 84 s (event 1: ~2 h, StopOnError was still 1). We found no loss of stored data. Hypothesis (source reading, unproven): insert-path var-page reorganisation (alloc_var_part -> Tup_varsize_page::alloc_record -> reorg) runs without exclusive fragment access, while query-thread readers dereference var-part pointers after releasing the TUP fragment mutex. See "Suggested fix" for file:line references. Attachment ndb-varsize-assert-bundle.tgz (README.txt inside): per event the error log, an out.log excerpt, the crashing query-thread and owning-LDM traces; config.ini; full table definitions; and the full-length versions of this description and of the suggested fix (full-description.txt, full-suggested-fix.txt). Everything is anonymised. 2. ENVIRONMENT mysql-cluster-community 8.4.11-1ubuntu24.04 (repo.mysql.com apt), Ubuntu 24.04, kernel 6.8.0. Hosts: single-socket AMD EPYC 4004-series (Zen 4), 16C/32T, 2 x 32 GB DDR5 (no ECC beyond DDR5 on-die ECC), NVMe. 6 data nodes, NoOfReplicas=2, NG0={1,2} NG1={3,4} NG2={5,6}; 3 management nodes; one SQL node per data node host (ndb_cluster_connection_pool=4). AutomaticThreadConfig=1, the same on all 6 data nodes: NumCPUs=16 until a rolling restart on 2026-09-03 (events 1-2: 4 ldm, 4 query = DBQLQH 1-4), NumCPUs=24 since then (event 3: 6 ldm thr 2-7 = DBLQH 1-6, 6 query thr 8-13 = DBQLQH 1-6). DataMemory=32G (40G at event 1), TransactionMemory=3G, LockPagesInMainMemory=1, StopOnError=0 (1 at event 1), MaxNoOfConcurrentOperations=262144, MaxNoOfConcurrentScans=500. SQL nodes: READ-COMMITTED, ndb_batch_size=1M, ndb_read_backup=ON. 3. THE THREE EVENTS ("35/3" = table 35 fragment 3) Event 1 Event 2 Event 3 Time (UTC) 2026-08-31 13:58:44 2026-09-03 21:11:23 2026-10-17 23:24:36 Node / thread 4 / thr 9 DBQLQH(4) 4 / thr 8 DBQLQH(3) 2 / thr 8 DBQLQH(1) Read ordered-index scan PK read: LQHKEYREQ ordered-index range DESC, no bound, Read Simple Dirty scan ASC; died on 1st pushed condition ACC_CHECK_SCAN after its SCAN_FRAGREQ Object index 142 (idx_ts), table 107, 107/4 index 141 (PRIMARY), 35/10 35/3 Requester API 154 via DBTC(2) API 160 via DBTC(1) API 177 via DBTC(2) on node 2 on node 4 (local) on node 5 Abort path interpreterNextLab > handleReadReq > interpreterStartLab > readAttributes > readAttributes > readAttributes > varsize_reader read_packed > read_packed > varsize_reader varsize_reader Owning LDM thr 5 = DBLQH(4): thr 3 = DBLQH(2): thr 7 = DBLQH(6): Write 35/10 -> ACC Insert 107/4: Write 35/3 -> ACC insert; alloc_fix_rec alloc_fix_rec + insert; alloc_fix_rec + alloc_var_part alloc_var_part + alloc_var_part (existing page); then (existing page); then (existing page); then waits at DblqhMain COMMIT waits at waits at DblqhMain 7105 (write-key) DblqhMain 7209 7105 (write-key) Caveats: (a) the crashing node always held the primary replica; with read_backup=1 that is just DBTC read routing here, not a clue. (b) A stuck owning LDM is also the expected consequence of any query-thread crash (the dead thread never releases fragment access), so it fits a race but does not prove one. 4. OUT.LOG (full stacks and base-relative offsets of all three events in the attachment) Event 3 stack: varsize_reader < read_packed < readAttributes < interpreterStartLab < execTUPKEYREQ < Dblqh::nextScanConfScanLab < continue_next_scan_conf < Dbtux::continue_scan < Dblqh::execACC_CHECK_SCAN. All three error logs: Error 6000, "Signal 6 received; Aborted". 5. TABLES (anonymised; full DDL of both tables in the attachment) Table id 35 (t1; indexes 141 = PRIMARY, 142 = idx_ts, 86 = idx_k1_ts), 13 columns: PRIMARY KEY (k1 varchar(12) latin1, k2 char(32) latin1), PARTITION BY KEY (k1), utf8mb4 default; var-size: k1, s1 + s2 varchar(255) COLUMN_FORMAT DYNAMIC, s3 varchar(45), s4 varchar(32), s5 varchar(20) (static); fixed: ts int (idx_ts, idx_k1_ts = (k1,ts)) and small int/decimal/char(2) columns. NDB_TABLE=NOLOGGING=1. Table id 107 (t2; index 114 = PRIMARY): same PK and partitioning, 28 columns; varchar s1..s14, of which s1, s2, s4 (varchar(255)) and s9 (varchar(64)) are DYNAMIC, declared at CREATE TABLE on 2026-08-29. Both tables: NOLOGGING, Row Checksum 1, ForceVarPart 1, FragmentCount 12, FOR_RP_BY_LDM, read_backup 1, no BLOB/TEXT/JSON; each mixes static var-size and DYNAMIC columns. 6. WORKLOAD (high-churn session-style tables) t1: several hundred thousand rows. All 6 SQL nodes write it with multi-row REPLACE (~1000 rows per statement, on the order of 100k rows/min); a REPLACE of a missing key becomes an insert. Rows expire a few minutes after their last write; once a minute one SQL node deletes them in chunks (READ COMMITTED ordered read on idx_ts, then a range DELETE on ts). Readers are READ COMMITTED per-k1 queries (e.g. "WHERE k1 = ? ORDER BY ts DESC LIMIT n"), which DBTC can route to query threads. t2 (tens of thousands of rows): single-row PK reads and writes. The statements behind the crashing reads were not identified. 7. RULED OUT - OOM / memory pressure, watchdog / overload / GCP warnings: none before any abort. - Index-statistics scans: stat 0, API requesters, and none running at event 3. - One table / index / node: 2 tables, 3 access paths (141 ASC, 142 DESC, hash PK), 2 nodes. - Hardware (unlikely, not excluded): no MCE/EDAC/AER events, HardwareCorrupted 0 kB on all 6 hosts; different DIMM makers on nodes 2 and 4; node 4 quiet for 44 days after 2 events in 3 days; Row Checksum covers the fixed part incl. the var part reference. But the RAM is non-ECC, so a bit flip in any one event cannot be strictly excluded. - Changes: no package or config change on the day of event 3. 8. IMPACT The node was rebuilt from its partner; SQL clients stalled ~6 s until the node was declared dead, and in-flight transactions got temporary errors (4010, 4028, 4119, 270). If the race hypothesis is right, some reads could also return wrong var-size values with no error instead of tripping the require; we have no evidence either way. How to repeat: We cannot reproduce this on demand; it is not deterministic. Below are the conditions under which it happened, and a stress setup that should make the suspected race more likely. Observed rate: 3 events in about 350 data-node-days (6 data nodes, about two months of production): 2026-08-31, 2026-09-03 and 2026-10-17. The events hit two different nodes. The thread layout was NumCPUs=16 for the first two and NumCPUs=24 for the third. Conditions in production (DDL and settings are in the Description): 1. ndbmtd with query threads: AutomaticThreadConfig=1, one query thread per LDM thread, NoOfReplicas=2, read_backup=1, 12 fragments per table (FOR_RP_BY_LDM). 2. NOLOGGING tables with a 2-column PK (k1 varchar(12), k2 char(32)), PARTITION BY KEY (k1), extra ordered indexes (t1: on (ts) and, from after event 1, on (k1, ts); t2: on (n3, ts)), and several var-size columns: static varchar columns plus COLUMN_FORMAT DYNAMIC varchar(255) columns whose values vary in length. 3. Writers (all 6 SQL nodes): continuous multi-row REPLACE of about 1000 rows per statement, on the order of 100k rows/min. A REPLACE of a key that is not present becomes an INSERT, which allocates a var part (alloc_var_part). On the second table: single-row writes, e.g. the INSERT of a new key in event 2. Its expiry job also runs an UPDATE (by PK list) that sets a DYNAMIC varchar column to ''. 4. Expiry: every minute, rows older than a few minutes are deleted in chunks of 10000 rows (an ordered READ COMMITTED snapshot via the (ts) index, then a range DELETE on ts). As a result, var pages are constantly freed and re-filled, so free space inside them is fragmented. 5. Readers: many concurrent READ COMMITTED statements, which DBTC can route to query threads, reading var-size columns of the same fragments: a. primary-key lookups (WHERE k1 = ? AND k2 = ?): the access type of event 2 b. single-partition range scans on PRIMARY (WHERE k1 = ?): the access type of event 3 c. ordered scans with a pushed condition on a var-size column, e.g. WHERE k1 = ? ORDER BY ts DESC LIMIT n through the (ts) index: the access type of event 1 The actual statements were not identified. Suggested stress setup (we cannot run debug builds in production): - Concentrate the race on few fragments: 1-2 LDM threads with the maximum number of query threads (query = 2x or 3x ldm). Use a single-fragment table, or make all readers and writers hit one partition key range. - Make Tup_varsize_page::reorg() frequent. Insert rows whose var parts vary strongly in size (0-255 bytes in several columns), delete a random half, and keep inserting. That way alloc_record() often finds enough total free space but not enough contiguous space (alloc_size >= largest_size). - Run 16-64 client threads doing READ COMMITTED PK reads and k1-range scans that return all var-size columns, concurrently with 4-8 threads doing INSERT/REPLACE of new keys plus DELETEs. - Make silent corruption detectable: store each var column as a deterministic function of (k1, k2), e.g. a repeated hash of length f(k1,k2). Every reader then checks what it receives. If the hypothesis is right, mismatches may also appear, possibly before the require fires. - In a debug build: add a jam()/counter to Tup_varsize_page::reorg(), enable the #if 0 consistency check in varsize_reader (DbtupRoutines.cpp:673-692), and if possible delay query-thread reads between DbtupExecQuery.cpp:1209 (mutex released) and readAttributes() to widen the window. Expected result if the hypothesis holds: DbtupRoutines.cpp:666 require((srcBytes <= max_var_size)) in a DBQLQH thread, and/or reads returning var-column values that belong to other rows. Suggested fix: HYPOTHESIS FROM SOURCE READING, NOT A VERIFIED ROOT CAUSE. (Longer version with quoted code: full-suggested-fix.txt in the attachment.) References are to tag mysql-cluster-8.4.11, storage/ndb/src/kernel/blocks/dbtup/ unless another path is given. We have not reproduced the crash; a release build cannot show whether a page reorg happened (tuppage.cpp has no jam()). By our diff, DbtupRoutines.cpp, DbtupExecQuery.cpp, DbtupVarAlloc.cpp, DbtupCommit.cpp, Dbtup.hpp, Dblqh.hpp and DblqhMain.cpp are unchanged in mysql-cluster-26.7.0 and the 9.7 branch. A. The failing check DbtupRoutines.cpp:666 require(srcBytes <= max_var_size) in varsize_reader. srcBytes is the difference of two Uint16 entries of the row's offset array, read directly from the var page (readVarSizeNotNULL 791-795; readDynVarSizeShrunkenNotNULL 1319-1321). If those bytes no longer hold this row's offset array, the difference can exceed the column maximum (trips the require) or come out small and plausible (silently returns another row's bytes). B. Query-thread readers do not hold the mutex while they read DbtupExecQuery.cpp execTUPKEYREQ, ZREAD: 1183 acquire_frag_mutex_read; 1192 setup_read -> prepare_read stores pointers INTO the var page (4458, 4491-4492: m_data_ptr, m_offset_array_ptr); 1199-1208 checksum under the mutex; 1209 release_frag_mutex_read; 1211 handleReadReq -> readAttributes (or interpreterStartLab). Dbtup.hpp:856-867: in query threads the mutex is tup_frag_mutex[logicalPageId & 3], hashed by the FIXED-part page. 4850-4855 states the assumption that holding the mutex only during prepare_read is enough. C. The insert path can move other rows' var parts without exclusive fragment access handleInsertReq (DbtupExecQuery.cpp:2458-2461) -> alloc_var_rec (DbtupVarAlloc.cpp:83-115) -> alloc_fix_rec (takes only the mutex of the NEW row's fixed page, DbtupFixAlloc.cpp:128) -> alloc_var_part (117-157): get_alloc_page returns an existing var page (jam 138, present in all three LDM JAMs), then Var_page::alloc_record (147-149). tuppage.cpp:239-251 alloc_record: if (alloc_size >= largest_size) reorg(temp); reorg (368-396) re-packs every live entry on the page, so the var parts of OTHER rows can move. No upgrade_to_exclusive_frag_access() is taken on this path ("An initial INSERT operation requires no mutex", DbtupExecQuery.cpp:1349-1355); fragment access is only upgraded later (1362-1373, write-key, before the TUX insert), which is where the LDM was blocked in events 1 and 3 (DblqhMain.cpp:7103-7105). alloc_var_rowid (DbtupVarAlloc.cpp:563-590) is similar. D. Contrast: other paths that move var data do take exclusive access handle_size_change_after_update (DbtupExecQuery.cpp:4796-4857): its comment says var-part reallocation "is intruding on all readers from the query thread ... it can even reorganise an entire page", and it calls upgrade_to_exclusive_frag_access (4825) around realloc_var_part (which can reorg or call alloc_var_part). Commit and dealloc are exclusive as well (DbtupCommit.cpp:1782, 201). The insert path in C reaches alloc_var_part -> alloc_record -> reorg without that protection. E. move_var_part (OPTIMIZE TABLE path) appears to have the same gap DbtupVarAlloc.cpp:365-367 calls alloc_record on the destination page (may reorg) before the upgrade at 393, although its own comment (382-392) says all places that reorganise pages and rows must use exclusive fragment access because query threads may be reading. F. Fit with the events, and what does not prove it - All three are query-thread reads through readAttributes -> varsize_reader on fragments where the owning LDM's JAM shows alloc_var_part on an existing var page. - A write prepare sets m_cond_write_key_waiters (Dblqh.hpp:5030) until write-key access is granted (DblqhMain.cpp:7118), and a resuming scan cannot re-acquire access while it is set (Dblqh.hpp:4867-4874). So in events 1 and 3 the blocked insert started after the scan last resumed: during the failing row read (the race) or after the require fired. - Not discriminating: allocations into existing var pages are common (18 in the event-3 LDM JAMs), and a stuck owning LDM is also the expected consequence of any query-thread crash. G. Possible fixes (suggestions only) 1. In alloc_var_part / alloc_record: when the allocation would reorg(), pick another or a fresh var page, or take exclusive fragment access first as handle_size_change_after_update does. alloc_var_rec calls alloc_var_part while holding the mutex from alloc_fix_rec, and DbtupExecQuery.cpp:1368-1371 warns against upgrading while holding a fragment mutex, so the decision probably has to be made before alloc_fix_rec, or the page choice has to avoid reorg() altogether. 2. In move_var_part: upgrade_to_exclusive_frag_access before alloc_record on the destination. 3. Diagnostics: a jam() in Tup_varsize_page::reorg, and the disabled #if 0 consistency check in varsize_reader (DbtupRoutines.cpp:673-692). H. Questions 1. Is this known internally? We found no public bug. Related but already fixed in our version: Bug #111932 / #35660890 (query-thread multi-range scan lost its fragment lock, 8.4.4). 2. If our reading is wrong, what keeps query threads off a fragment while its LDM runs alloc_var_part? 3. Is disabling query threads (AutomaticThreadConfig=0, ClassicFragmentation=false, explicit ThreadConfig with ldm={count=6}, query={count=0}) a supported and effective workaround? 4. Is a rolling restart that temporarily mixes nodes with and without query threads safe in 8.4.11 (remote DBTCs address DBQLQH instances directly)? 5. Should OPTIMIZE TABLE on tables with var-size columns be avoided until a fix (section E)?