Description:
In MySQL 8.0.40, dropping a FULLTEXT index can race with a concurrent INFORMATION_SCHEMA.INNODB_TABLES scan that opens InnoDB FTS auxiliary tables.
The problem is that ALTER TABLE ... DROP KEY/DROP INDEX on the parent table holds MDL for the parent table, e.g. test.t1, but the FTS auxiliary tables are separate DD/InnoDB tables with separate MDL keys, e.g. test/fts_..._index_*, test/fts_..._deleted, test/fts_..._being_deleted, etc.
INFORMATION_SCHEMA.INNODB_TABLES is filled by scanning mysql.tables and opening each InnoDB table through dd_process_dd_tables_rec_and_mtr_commit(), which calls dd_table_open_on_id(). FTS auxiliary tables are also persistent InnoDB DD tables, so this path can open an FTS auxiliary table and temporarily hold its Shared MDL and dict_table_t reference.
At the same time, DROP FULLTEXT INDEX drops the FTS auxiliary tables through fts_drop_table(). That function first opens the auxiliary table with dd_table_open_on_name() and a Shared MDL, but immediately closes it and releases the MDL before calling row_drop_table_for_mysql() to actually drop the auxiliary table.
Therefore the real drop of the FTS auxiliary table is not protected by an exclusive MDL on the auxiliary table name. If another thread still holds an open dict_table_t reference to the auxiliary table, row_drop_table_for_mysql() sees table->get_ref_count() > 0. In debug builds this reaches the debug assertion/error path in row_drop_table_for_mysql().
This is not prevented by the parent table MDL because the parent table and the FTS auxiliary tables use different MDL keys.
Relevant code paths in 8.0.40:
1. INFORMATION_SCHEMA.INNODB_TABLES opens auxiliary table:
storage/innobase/handler/i_s.cc
i_s_innodb_tables_fill_table()
-> dd_startscan_system(... mysql.tables ...)
-> dd_process_dd_tables_rec_and_mtr_commit(...)
-> dd_table_open_on_id(table_id, thd, mdl, true, false)
2. DROP FULLTEXT INDEX drops auxiliary table:
storage/innobase/fts/fts0fts.cc
fts_drop_index()
-> fts_drop_tables()/fts_drop_index_tables()
-> fts_drop_table(aux_table_name)
-> dd_table_open_on_name(thd, &mdl, aux_table_name, ...)
-> dd_table_close(table, thd, &mdl, true)
-> row_drop_table_for_mysql(aux_table_name, trx, false, nullptr)
3. Crash/assert point:
storage/innobase/row/row0mysql.cc
row_drop_table_for_mysql()
-> detects table->get_ref_count() > 0 while dropping the auxiliary table
How to repeat:
The race can be reproduced deterministically with a DEBUG_SYNC hook added to INFORMATION_SCHEMA.INNODB_TABLES after it opens an FTS auxiliary table and before it closes the table handle.
Patch used for deterministic reproduction:
--- storage/innobase/handler/i_s.cc
+++ storage/innobase/handler/i_s.cc
@@ -5437,6 +5437,14 @@ static int i_s_innodb_tables_fill_table(THD *thd, Table_ref *tables, Item *) {
dict_sys_mutex_exit();
if (table_rec != nullptr) {
i_s_dict_fill_innodb_tables(thd, table_rec, tables->table);
+
+ /* Test hook: stop after INFORMATION_SCHEMA.INNODB_TABLES has opened an
+ FTS auxiliary table and before it releases the table's Shared MDL and
+ dict_table_t reference. This makes DROP FULLTEXT INDEX races
+ deterministic in MTR. */
+ if (strstr(table_rec->name.m_name, "/fts_") != nullptr) {
+ DEBUG_SYNC_C("i_s_innodb_tables_holding_fts_aux");
+ }
}
Then run this MTR test:
--source include/have_debug_sync.inc
--source include/not_valgrind.inc
--source include/count_sessions.inc
--echo # Verify DROP FULLTEXT INDEX waits for I_S readers of FTS auxiliary tables.
--disable_query_log
--disable_result_log
CREATE TABLE t1 (
FTS_DOC_ID BIGINT UNSIGNED AUTO_INCREMENT NOT NULL,
id INT,
title VARCHAR(200),
PRIMARY KEY(FTS_DOC_ID),
FULLTEXT fidx(title)
) ENGINE=InnoDB;
INSERT INTO t1(id, title)
VALUES (1, 'mysql tutorial'), (2, 'innodb full text index');
connect (con1,localhost,root,,);
connect (con2,localhost,root,,);
connection con1;
SET DEBUG_SYNC =
'i_s_innodb_tables_holding_fts_aux SIGNAL is_holding WAIT_FOR release_reader';
--send SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLES WHERE NAME LIKE 'test/fts_%'
connection default;
SET DEBUG_SYNC = 'now WAIT_FOR is_holding';
connection con2;
--send ALTER TABLE t1 DROP KEY fidx
connection default;
# Give the DDL connection a chance to reach the FTS auxiliary table drop while
# con1 still holds the aux table dict_table_t reference from the I_S scan.
--sleep 1
SET DEBUG_SYNC = 'now SIGNAL release_reader';
connection con1;
--reap
disconnect con1;
connection con2;
--reap
disconnect con2;
connection default;
SET DEBUG_SYNC = 'RESET';
DROP TABLE t1;
--enable_result_log
--enable_query_log
--echo # Success
On an unpatched debug build, the server aborts in row_drop_table_for_mysql() when DROP FULLTEXT INDEX tries to drop an FTS auxiliary table while the I_S scan still holds an open dict_table_t reference.
Observed stack shape:
row_drop_table_for_mysql()
fts_drop_table()
fts_drop_tables()/fts_drop_index_tables()
fts_drop_index()
commit_cache_norebuild()
ha_innobase::commit_inplace_alter_table_impl()
ha_innobase::commit_inplace_alter_table()
mysql_inplace_alter_table()
mysql_alter_table()
ALTER TABLE t1 DROP KEY fidx
Suggested fix:
When dropping FTS auxiliary tables as part of DROP FULLTEXT INDEX, InnoDB should hold a metadata lock that actually protects the auxiliary table name for the whole drop operation.
Possible fix directions:
1. Acquire an exclusive MDL on each FTS auxiliary table before calling row_drop_table_for_mysql(), and keep it until the auxiliary table has been removed from the InnoDB dictionary/DD/cache.
2. Avoid the current pattern in fts_drop_table() where the auxiliary table is opened with Shared MDL, closed, and only then passed to row_drop_table_for_mysql() for the actual drop.
3. Ensure INFORMATION_SCHEMA.INNODB_TABLES and other DD readers that open FTS auxiliary tables by table_id cannot keep an auxiliary table dict_table_t reference concurrently with its physical/logical drop.
The parent table MDL is not sufficient because the parent table and FTS auxiliary tables have different MDL keys.
Description: In MySQL 8.0.40, dropping a FULLTEXT index can race with a concurrent INFORMATION_SCHEMA.INNODB_TABLES scan that opens InnoDB FTS auxiliary tables. The problem is that ALTER TABLE ... DROP KEY/DROP INDEX on the parent table holds MDL for the parent table, e.g. test.t1, but the FTS auxiliary tables are separate DD/InnoDB tables with separate MDL keys, e.g. test/fts_..._index_*, test/fts_..._deleted, test/fts_..._being_deleted, etc. INFORMATION_SCHEMA.INNODB_TABLES is filled by scanning mysql.tables and opening each InnoDB table through dd_process_dd_tables_rec_and_mtr_commit(), which calls dd_table_open_on_id(). FTS auxiliary tables are also persistent InnoDB DD tables, so this path can open an FTS auxiliary table and temporarily hold its Shared MDL and dict_table_t reference. At the same time, DROP FULLTEXT INDEX drops the FTS auxiliary tables through fts_drop_table(). That function first opens the auxiliary table with dd_table_open_on_name() and a Shared MDL, but immediately closes it and releases the MDL before calling row_drop_table_for_mysql() to actually drop the auxiliary table. Therefore the real drop of the FTS auxiliary table is not protected by an exclusive MDL on the auxiliary table name. If another thread still holds an open dict_table_t reference to the auxiliary table, row_drop_table_for_mysql() sees table->get_ref_count() > 0. In debug builds this reaches the debug assertion/error path in row_drop_table_for_mysql(). This is not prevented by the parent table MDL because the parent table and the FTS auxiliary tables use different MDL keys. Relevant code paths in 8.0.40: 1. INFORMATION_SCHEMA.INNODB_TABLES opens auxiliary table: storage/innobase/handler/i_s.cc i_s_innodb_tables_fill_table() -> dd_startscan_system(... mysql.tables ...) -> dd_process_dd_tables_rec_and_mtr_commit(...) -> dd_table_open_on_id(table_id, thd, mdl, true, false) 2. DROP FULLTEXT INDEX drops auxiliary table: storage/innobase/fts/fts0fts.cc fts_drop_index() -> fts_drop_tables()/fts_drop_index_tables() -> fts_drop_table(aux_table_name) -> dd_table_open_on_name(thd, &mdl, aux_table_name, ...) -> dd_table_close(table, thd, &mdl, true) -> row_drop_table_for_mysql(aux_table_name, trx, false, nullptr) 3. Crash/assert point: storage/innobase/row/row0mysql.cc row_drop_table_for_mysql() -> detects table->get_ref_count() > 0 while dropping the auxiliary table How to repeat: The race can be reproduced deterministically with a DEBUG_SYNC hook added to INFORMATION_SCHEMA.INNODB_TABLES after it opens an FTS auxiliary table and before it closes the table handle. Patch used for deterministic reproduction: --- storage/innobase/handler/i_s.cc +++ storage/innobase/handler/i_s.cc @@ -5437,6 +5437,14 @@ static int i_s_innodb_tables_fill_table(THD *thd, Table_ref *tables, Item *) { dict_sys_mutex_exit(); if (table_rec != nullptr) { i_s_dict_fill_innodb_tables(thd, table_rec, tables->table); + + /* Test hook: stop after INFORMATION_SCHEMA.INNODB_TABLES has opened an + FTS auxiliary table and before it releases the table's Shared MDL and + dict_table_t reference. This makes DROP FULLTEXT INDEX races + deterministic in MTR. */ + if (strstr(table_rec->name.m_name, "/fts_") != nullptr) { + DEBUG_SYNC_C("i_s_innodb_tables_holding_fts_aux"); + } } Then run this MTR test: --source include/have_debug_sync.inc --source include/not_valgrind.inc --source include/count_sessions.inc --echo # Verify DROP FULLTEXT INDEX waits for I_S readers of FTS auxiliary tables. --disable_query_log --disable_result_log CREATE TABLE t1 ( FTS_DOC_ID BIGINT UNSIGNED AUTO_INCREMENT NOT NULL, id INT, title VARCHAR(200), PRIMARY KEY(FTS_DOC_ID), FULLTEXT fidx(title) ) ENGINE=InnoDB; INSERT INTO t1(id, title) VALUES (1, 'mysql tutorial'), (2, 'innodb full text index'); connect (con1,localhost,root,,); connect (con2,localhost,root,,); connection con1; SET DEBUG_SYNC = 'i_s_innodb_tables_holding_fts_aux SIGNAL is_holding WAIT_FOR release_reader'; --send SELECT NAME FROM INFORMATION_SCHEMA.INNODB_TABLES WHERE NAME LIKE 'test/fts_%' connection default; SET DEBUG_SYNC = 'now WAIT_FOR is_holding'; connection con2; --send ALTER TABLE t1 DROP KEY fidx connection default; # Give the DDL connection a chance to reach the FTS auxiliary table drop while # con1 still holds the aux table dict_table_t reference from the I_S scan. --sleep 1 SET DEBUG_SYNC = 'now SIGNAL release_reader'; connection con1; --reap disconnect con1; connection con2; --reap disconnect con2; connection default; SET DEBUG_SYNC = 'RESET'; DROP TABLE t1; --enable_result_log --enable_query_log --echo # Success On an unpatched debug build, the server aborts in row_drop_table_for_mysql() when DROP FULLTEXT INDEX tries to drop an FTS auxiliary table while the I_S scan still holds an open dict_table_t reference. Observed stack shape: row_drop_table_for_mysql() fts_drop_table() fts_drop_tables()/fts_drop_index_tables() fts_drop_index() commit_cache_norebuild() ha_innobase::commit_inplace_alter_table_impl() ha_innobase::commit_inplace_alter_table() mysql_inplace_alter_table() mysql_alter_table() ALTER TABLE t1 DROP KEY fidx Suggested fix: When dropping FTS auxiliary tables as part of DROP FULLTEXT INDEX, InnoDB should hold a metadata lock that actually protects the auxiliary table name for the whole drop operation. Possible fix directions: 1. Acquire an exclusive MDL on each FTS auxiliary table before calling row_drop_table_for_mysql(), and keep it until the auxiliary table has been removed from the InnoDB dictionary/DD/cache. 2. Avoid the current pattern in fts_drop_table() where the auxiliary table is opened with Shared MDL, closed, and only then passed to row_drop_table_for_mysql() for the actual drop. 3. Ensure INFORMATION_SCHEMA.INNODB_TABLES and other DD readers that open FTS auxiliary tables by table_id cannot keep an auxiliary table dict_table_t reference concurrently with its physical/logical drop. The parent table MDL is not sufficient because the parent table and FTS auxiliary tables have different MDL keys.