Description:
## Description
A standby/replica server crashes intermittently under concurrent DDL replay (e.g. `DROP TABLE` replayed by the SQL thread) while a concurrent information-schema statistics query is running. The crash dereferences an empty `dd::Collection<dd::Index*>` inside `dd_first_index`, i.e. the cached `dd::Table` object observed by the query has an empty `indexes()` collection even though the persisted `dd::Table` for an InnoDB table can never have an empty index set (InnoDB always materializes a clustered index, implicit `DB_ROW_ID` when no explicit primary key is given).
Tool / reproducer: sqlancer-2.0.0 .
The crash is a concurrency/timing defect: the invalidation of the DD cache object by the replicating DDL and the acquisition/use of that object by the I_S query are not strictly serialized, so the query can observe an intermediate-state cached `dd::Table` whose `indexes()` is momentarily empty. Adding an empty-collection guard inside `dd_first` / `dd_first_index` is **not** the correct fix — it merely downgrades the crash to a silent wrong result (e.g. cardinality = `4294967295` or "table not found") and masks the root cause.
Crash stack:
```
#0 dd::Collection<dd::Index*>::Collection_const_iterator::operator* (this=0x7f2a504eb350) at ../../sql/dd/collection.cc:76
#1 dd_first<dd::Table, dd::Index> (table=...) at ../../../storage/innobase/include/dict0dd.h:680
#2 dd_first_index (table=...) at ../../../storage/innobase/dict/dict0dd.cc:3773
#3 dd_open_table_one<dd::Table> (client=..., table=..., norm_name="database74/tb1_990", dd_table=..., thd=..., fk_list=...) at ../../../storage/innobase/dict/dict0dd.cc:5155
#4 dd_open_table<dd::Table> (client=..., table=..., norm_name="database74/tb1_990", dd_table=..., thd=...) at ../../../storage/innobase/dict/dict0dd.cc (stl_deque.h:831)
#5 dd_table_open_on_dd_obj (thd=..., client=..., dd_table=..., dd_part=0x0, tbl_name=..., table=@0x7f2a504ecb48: 0x0, td=0x0) at ../../../storage/innobase/dict/dict0dd.cc:477
#6 dd_table_open_on_name (thd=..., mdl=0x7f2a504ecc38, name="database74/tb1_990", dict_locked=..., ignore_err=0, error=0x0) at ../../../storage/innobase/dict/dict0dd.cc:1077
#7 innobase_get_index_column_cardinality(...) [clone .cold.0] () at ../../../storage/innobase/handler/ha_innodb.cc:19690
#8 dd::info_schema::Table_statistics::read_stat_from_SE (this=..., thd=..., schema_name_ptr=..., table_name_ptr=..., index_name_ptr=..., column_name_ptr=..., index_ordinal_position=1, column_ordinal_position=0, se_private_id=403240, ts_se_private_data=0x0, tbl_se_private_data=0x0, stype=dd::info_schema::INDEX_COLUMN_CARDINALITY, hton=...) at ../../sql/dd/info_schema/table_stats.cc:674
#9 dd::info_schema::Table_statistics::read_stat (this=..., thd=..., schema_name_ptr=..., table_name_ptr=..., index_name_ptr=..., partition_name=..., column_name_ptr=..., index_ordinal_position=1, column_ordinal_position=0, engine_name_ptr=..., se_private_id=403240, ts_se_private_data=0x0, tbl_se_private_data=0x0, table_stat_data=@..., cached_timestamp=@..., stype=dd::info_schema::INDEX_COLUMN_CARDINALITY) at ../../sql/dd/info_schema/table_stats.cc:512
#10 Item_func_internal_index_column_cardinality::val_int (this=...) at ../../sql/item_func.cc:10243
#11 Item::save_in_field_inner (this=..., field=..., no_conversions=...) at ../../sql/item.cc:6798
...
#14 MaterializeIterator<DummyIteratorProfiler>::MaterializeQueryBlock (this=..., query_block=..., stored_rows=...) at ../../sql/iterators/composite_iterators.cc:1467
...
```
How to repeat:
1. Deploy a source/replica topology (InnoDB). On the replica, run the SQL thread for DDL replay.
2. Drive concurrent load on the replica with sqlancer-2.0.0:
```
java -jar target/sqlancer-2.0.0.jar \
--username <user> --password <pwd> --host <host> --port <port> \
--num-tries=99999999 --num-threads 256 --max-tables 500 \
--test-tablespace true mysql \
--oracle FUZZER --oracle TLP_WHERE \
--mysql-bombard true --mysql-bombard-workers 4 \
> sqlancer_output.log 2>&1
```
3. After running for some time, the replica crashes with a backtrace whose top frame is `dd::Collection<dd::Index*>::Collection_const_iterator::operator*` inside `dd_first_index` (`storage/innobase/dict/dict0dd.cc:3773`), reached via `innobase_get_index_column_cardinality` (`storage/innobase/handler/ha_innodb.cc:19690`) from `dd::info_schema::Table_statistics::read_stat_from_SE` (`sql/dd/info_schema/table_stats.cc:674`).
The failure is probabilistic (requires the DROP-replay invalidation to land inside the query's acquire/use window), so it needs sustained concurrent pressure to reproduce.
A deterministic reproduction approach: use an MTR `DEBUG_SYNC` point at `before_dd_first_index_in_dd_open_table_one` to pause the query, trigger a `DROP TABLE` replay on the replica at the same moment, and inspect whether the query's `dd_table` came from a cache hit or an uncached direct read and whether `indexes()` is empty at that instant — this pinpoints the exact fix site.
Suggested fix:
In `innobase_get_index_column_cardinality` → `dd_table_open_on_name`, after obtaining the cached `dd_table` and **before** dereferencing `->indexes()`, re-validate against a self-consistent snapshot fetched with `acquire_uncached` from the persistence layer (whose `indexes()` is guaranteed non-empty). This bypasses the intermediate-state window of the cached object, eliminating the concurrency window in which the query can observe a cached `dd::Table` whose `indexes()` is momentarily empty. This mirrors the previously-accepted fix pattern for the analogous `I_S.files` replica crash (acquire an `MDL_SHARE` lock, then re-validate via `acquire_uncached`), rather than touching the replay concurrency model.
Description: ## Description A standby/replica server crashes intermittently under concurrent DDL replay (e.g. `DROP TABLE` replayed by the SQL thread) while a concurrent information-schema statistics query is running. The crash dereferences an empty `dd::Collection<dd::Index*>` inside `dd_first_index`, i.e. the cached `dd::Table` object observed by the query has an empty `indexes()` collection even though the persisted `dd::Table` for an InnoDB table can never have an empty index set (InnoDB always materializes a clustered index, implicit `DB_ROW_ID` when no explicit primary key is given). Tool / reproducer: sqlancer-2.0.0 . The crash is a concurrency/timing defect: the invalidation of the DD cache object by the replicating DDL and the acquisition/use of that object by the I_S query are not strictly serialized, so the query can observe an intermediate-state cached `dd::Table` whose `indexes()` is momentarily empty. Adding an empty-collection guard inside `dd_first` / `dd_first_index` is **not** the correct fix — it merely downgrades the crash to a silent wrong result (e.g. cardinality = `4294967295` or "table not found") and masks the root cause. Crash stack: ``` #0 dd::Collection<dd::Index*>::Collection_const_iterator::operator* (this=0x7f2a504eb350) at ../../sql/dd/collection.cc:76 #1 dd_first<dd::Table, dd::Index> (table=...) at ../../../storage/innobase/include/dict0dd.h:680 #2 dd_first_index (table=...) at ../../../storage/innobase/dict/dict0dd.cc:3773 #3 dd_open_table_one<dd::Table> (client=..., table=..., norm_name="database74/tb1_990", dd_table=..., thd=..., fk_list=...) at ../../../storage/innobase/dict/dict0dd.cc:5155 #4 dd_open_table<dd::Table> (client=..., table=..., norm_name="database74/tb1_990", dd_table=..., thd=...) at ../../../storage/innobase/dict/dict0dd.cc (stl_deque.h:831) #5 dd_table_open_on_dd_obj (thd=..., client=..., dd_table=..., dd_part=0x0, tbl_name=..., table=@0x7f2a504ecb48: 0x0, td=0x0) at ../../../storage/innobase/dict/dict0dd.cc:477 #6 dd_table_open_on_name (thd=..., mdl=0x7f2a504ecc38, name="database74/tb1_990", dict_locked=..., ignore_err=0, error=0x0) at ../../../storage/innobase/dict/dict0dd.cc:1077 #7 innobase_get_index_column_cardinality(...) [clone .cold.0] () at ../../../storage/innobase/handler/ha_innodb.cc:19690 #8 dd::info_schema::Table_statistics::read_stat_from_SE (this=..., thd=..., schema_name_ptr=..., table_name_ptr=..., index_name_ptr=..., column_name_ptr=..., index_ordinal_position=1, column_ordinal_position=0, se_private_id=403240, ts_se_private_data=0x0, tbl_se_private_data=0x0, stype=dd::info_schema::INDEX_COLUMN_CARDINALITY, hton=...) at ../../sql/dd/info_schema/table_stats.cc:674 #9 dd::info_schema::Table_statistics::read_stat (this=..., thd=..., schema_name_ptr=..., table_name_ptr=..., index_name_ptr=..., partition_name=..., column_name_ptr=..., index_ordinal_position=1, column_ordinal_position=0, engine_name_ptr=..., se_private_id=403240, ts_se_private_data=0x0, tbl_se_private_data=0x0, table_stat_data=@..., cached_timestamp=@..., stype=dd::info_schema::INDEX_COLUMN_CARDINALITY) at ../../sql/dd/info_schema/table_stats.cc:512 #10 Item_func_internal_index_column_cardinality::val_int (this=...) at ../../sql/item_func.cc:10243 #11 Item::save_in_field_inner (this=..., field=..., no_conversions=...) at ../../sql/item.cc:6798 ... #14 MaterializeIterator<DummyIteratorProfiler>::MaterializeQueryBlock (this=..., query_block=..., stored_rows=...) at ../../sql/iterators/composite_iterators.cc:1467 ... ``` How to repeat: 1. Deploy a source/replica topology (InnoDB). On the replica, run the SQL thread for DDL replay. 2. Drive concurrent load on the replica with sqlancer-2.0.0: ``` java -jar target/sqlancer-2.0.0.jar \ --username <user> --password <pwd> --host <host> --port <port> \ --num-tries=99999999 --num-threads 256 --max-tables 500 \ --test-tablespace true mysql \ --oracle FUZZER --oracle TLP_WHERE \ --mysql-bombard true --mysql-bombard-workers 4 \ > sqlancer_output.log 2>&1 ``` 3. After running for some time, the replica crashes with a backtrace whose top frame is `dd::Collection<dd::Index*>::Collection_const_iterator::operator*` inside `dd_first_index` (`storage/innobase/dict/dict0dd.cc:3773`), reached via `innobase_get_index_column_cardinality` (`storage/innobase/handler/ha_innodb.cc:19690`) from `dd::info_schema::Table_statistics::read_stat_from_SE` (`sql/dd/info_schema/table_stats.cc:674`). The failure is probabilistic (requires the DROP-replay invalidation to land inside the query's acquire/use window), so it needs sustained concurrent pressure to reproduce. A deterministic reproduction approach: use an MTR `DEBUG_SYNC` point at `before_dd_first_index_in_dd_open_table_one` to pause the query, trigger a `DROP TABLE` replay on the replica at the same moment, and inspect whether the query's `dd_table` came from a cache hit or an uncached direct read and whether `indexes()` is empty at that instant — this pinpoints the exact fix site. Suggested fix: In `innobase_get_index_column_cardinality` → `dd_table_open_on_name`, after obtaining the cached `dd_table` and **before** dereferencing `->indexes()`, re-validate against a self-consistent snapshot fetched with `acquire_uncached` from the persistence layer (whose `indexes()` is guaranteed non-empty). This bypasses the intermediate-state window of the cached object, eliminating the concurrency window in which the query can observe a cached `dd::Table` whose `indexes()` is momentarily empty. This mirrors the previously-accepted fix pattern for the analogous `I_S.files` replica crash (acquire an `MDL_SHARE` lock, then re-validate via `acquire_uncached`), rather than touching the replay concurrency model.