Bug #121047 Replica crash in `dd_first_index` on empty `dd::Table::indexes()` collection under concurrent DD
Submitted: 1 Aug 1:37
Reporter: Alice Alice Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Server: Data Dictionary Severity:S2 (Serious)
Version:8.0 OS:Linux
Assigned to: CPU Architecture:x86

[1 Aug 1:37] Alice Alice
Description:
## Description

A standby/replica server crashes intermittently under concurrent DDL replay (e.g. `DROP TABLE` replayed by the SQL thread) while a concurrent information-schema statistics query is running. The crash dereferences an empty `dd::Collection<dd::Index*>` inside `dd_first_index`, i.e. the cached `dd::Table` object observed by the query has an empty `indexes()` collection even though the persisted `dd::Table` for an InnoDB table can never have an empty index set (InnoDB always materializes a clustered index, implicit `DB_ROW_ID` when no explicit primary key is given).

Tool / reproducer: sqlancer-2.0.0 .

The crash is a concurrency/timing defect: the invalidation of the DD cache object by the replicating DDL and the acquisition/use of that object by the I_S query are not strictly serialized, so the query can observe an intermediate-state cached `dd::Table` whose `indexes()` is momentarily empty. Adding an empty-collection guard inside `dd_first` / `dd_first_index` is **not** the correct fix — it merely downgrades the crash to a silent wrong result (e.g. cardinality = `4294967295` or "table not found") and masks the root cause.

Crash stack:

```
#0  dd::Collection<dd::Index*>::Collection_const_iterator::operator* (this=0x7f2a504eb350) at ../../sql/dd/collection.cc:76
#1  dd_first<dd::Table, dd::Index> (table=...) at ../../../storage/innobase/include/dict0dd.h:680
#2  dd_first_index (table=...) at ../../../storage/innobase/dict/dict0dd.cc:3773
#3  dd_open_table_one<dd::Table> (client=..., table=..., norm_name="database74/tb1_990", dd_table=..., thd=..., fk_list=...) at ../../../storage/innobase/dict/dict0dd.cc:5155
#4  dd_open_table<dd::Table> (client=..., table=..., norm_name="database74/tb1_990", dd_table=..., thd=...) at ../../../storage/innobase/dict/dict0dd.cc (stl_deque.h:831)
#5  dd_table_open_on_dd_obj (thd=..., client=..., dd_table=..., dd_part=0x0, tbl_name=..., table=@0x7f2a504ecb48: 0x0, td=0x0) at ../../../storage/innobase/dict/dict0dd.cc:477
#6  dd_table_open_on_name (thd=..., mdl=0x7f2a504ecc38, name="database74/tb1_990", dict_locked=..., ignore_err=0, error=0x0) at ../../../storage/innobase/dict/dict0dd.cc:1077
#7  innobase_get_index_column_cardinality(...) [clone .cold.0] () at ../../../storage/innobase/handler/ha_innodb.cc:19690
#8  dd::info_schema::Table_statistics::read_stat_from_SE (this=..., thd=..., schema_name_ptr=..., table_name_ptr=..., index_name_ptr=..., column_name_ptr=..., index_ordinal_position=1, column_ordinal_position=0, se_private_id=403240, ts_se_private_data=0x0, tbl_se_private_data=0x0, stype=dd::info_schema::INDEX_COLUMN_CARDINALITY, hton=...) at ../../sql/dd/info_schema/table_stats.cc:674
#9  dd::info_schema::Table_statistics::read_stat (this=..., thd=..., schema_name_ptr=..., table_name_ptr=..., index_name_ptr=..., partition_name=..., column_name_ptr=..., index_ordinal_position=1, column_ordinal_position=0, engine_name_ptr=..., se_private_id=403240, ts_se_private_data=0x0, tbl_se_private_data=0x0, table_stat_data=@..., cached_timestamp=@..., stype=dd::info_schema::INDEX_COLUMN_CARDINALITY) at ../../sql/dd/info_schema/table_stats.cc:512
#10 Item_func_internal_index_column_cardinality::val_int (this=...) at ../../sql/item_func.cc:10243
#11 Item::save_in_field_inner (this=..., field=..., no_conversions=...) at ../../sql/item.cc:6798
...
#14 MaterializeIterator<DummyIteratorProfiler>::MaterializeQueryBlock (this=..., query_block=..., stored_rows=...) at ../../sql/iterators/composite_iterators.cc:1467
...
```

How to repeat:
1. Deploy a source/replica topology (InnoDB). On the replica, run the SQL thread for DDL replay.
2. Drive concurrent load on the replica with sqlancer-2.0.0:

   ```
   java -jar target/sqlancer-2.0.0.jar \
     --username <user> --password <pwd> --host <host> --port <port> \
     --num-tries=99999999 --num-threads 256 --max-tables 500 \
     --test-tablespace true mysql \
     --oracle FUZZER --oracle TLP_WHERE \
     --mysql-bombard true --mysql-bombard-workers 4 \
     > sqlancer_output.log 2>&1
   ```
3. After running for some time, the replica crashes with a backtrace whose top frame is `dd::Collection<dd::Index*>::Collection_const_iterator::operator*` inside `dd_first_index` (`storage/innobase/dict/dict0dd.cc:3773`), reached via `innobase_get_index_column_cardinality` (`storage/innobase/handler/ha_innodb.cc:19690`) from `dd::info_schema::Table_statistics::read_stat_from_SE` (`sql/dd/info_schema/table_stats.cc:674`).

The failure is probabilistic (requires the DROP-replay invalidation to land inside the query's acquire/use window), so it needs sustained concurrent pressure to reproduce.

A deterministic reproduction approach: use an MTR `DEBUG_SYNC` point at `before_dd_first_index_in_dd_open_table_one` to pause the query, trigger a `DROP TABLE` replay on the replica at the same moment, and inspect whether the query's `dd_table` came from a cache hit or an uncached direct read and whether `indexes()` is empty at that instant — this pinpoints the exact fix site.

Suggested fix:
In `innobase_get_index_column_cardinality` → `dd_table_open_on_name`, after obtaining the cached `dd_table` and **before** dereferencing `->indexes()`, re-validate against a self-consistent snapshot fetched with `acquire_uncached` from the persistence layer (whose `indexes()` is guaranteed non-empty). This bypasses the intermediate-state window of the cached object, eliminating the concurrency window in which the query can observe a cached `dd::Table` whose `indexes()` is momentarily empty. This mirrors the previously-accepted fix pattern for the analogous `I_S.files` replica crash (acquire an `MDL_SHARE` lock, then re-validate via `acquire_uncached`), rather than touching the replay concurrency model.