| Bug #120561 | tablespace marked missing after rollback of copy-style ALTER | ||
|---|---|---|---|
| Submitted: | 28 May 11:11 | Modified: | 28 May 11:15 |
| Reporter: | Huaxiong Song (OCA) | Email Updates: | |
| Status: | Open | Impact on me: | |
| Category: | MySQL Server: DDL | Severity: | S3 (Non-critical) |
| Version: | 9.7/8.4/8.0 | OS: | Any |
| Assigned to: | CPU Architecture: | Any | |
[28 May 11:15]
Huaxiong Song
Sorry, the content under “Suggested fix” overlaps with “How to Repeat.” Let me reorganize it and send an updated version.
=========================================================================================================================
[How to repeat]
---------------
A debug-build reproduction is included below. The bug also surfaces in production builds whenever an atomic ALTER reaches err_with_mdl after both renames -- for example via deadlock during update_referencing_views_metadata(), a foreign-key parent invalidation failure, or commit failure -- but a deterministic non-debug repro requires precise concurrency, so the DBUG hook is added in sql/sql_table.cc purely to force the same code path that production failures take.
1. Build MySQL with -DWITH_DEBUG=1.
2. Add a DBUG hook in sql/sql_table.cc, in mysql_alter_table(), inside the
"if (!is_noop)" block immediately before the update_referencing_views_metadata()
call (this is the only post-rename point where err_with_mdl is still
reachable). The hook is added between the existing uncommitted_tables setup
and the existing update_referencing_views_metadata() call:
if (write_bin_log(thd, true, thd->query().str, thd->query().length,
atomic_replace && !is_noop))
goto err_with_mdl;
if (!is_noop) {
Uncommitted_tables_guard uncommitted_tables(thd);
uncommitted_tables.add_table(table_list);
/* >>> ADDED: force copy-style ALTER past both mysql_rename_table
calls to fail into err_with_mdl, triggering the rollback branch
in fil_op_replay_rename. <<< */
DBUG_EXECUTE_IF("force_alter_rollback_after_rename", {
my_error(ER_LOCK_DEADLOCK, MYF(0));
goto err_with_mdl;
});
if (update_referencing_views_metadata(thd, table_list, new_db, new_name,
!atomic_replace, &uncommitted_tables))
goto err_with_mdl;
if (alter_ctx.is_table_renamed())
tdc_remove_table(thd, TDC_RT_REMOVE_ALL, alter_ctx.new_db,
alter_ctx.new_name, false);
}
3. Run:
CREATE DATABASE `mydb-x`;
USE `mydb-x`;
CREATE TABLE t1 (id INT PRIMARY KEY, val VARCHAR(32)) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1, 'a'), (2, 'b'), (3, 'c');
FLUSH TABLES t1;
SET SESSION debug = '+d,force_alter_rollback_after_rename';
ALTER TABLE t1 FORCE, ALGORITHM=COPY;
-- ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
SET SESSION debug = '-d,force_alter_rollback_after_rename';
FLUSH TABLES t1;
SELECT * FROM t1;
-- ERROR 1812 (HY000): Tablespace is missing for table `mydb-x`.`t1`.
The .ibd file is fully present:
$ ls $datadir/mydb@002dx/
t1.ibd
Restarting the server makes the table readable again, which confirms the corruption is purely in the in-memory FIL cache, not on disk.
4. Equivalent reproductions for table-name, partition-name, and sub-partition-name variants by substituting:
- table: CREATE TABLE `t-name` in any database
- partition: PARTITION `p-low` VALUES LESS THAN (50)
- sub-partition: SUBPARTITION `sp-a`
In each case the .ibd on disk uses @002d encoding, while dd_load_tablespace looks up the system-charset form.
5. Tables/partitions whose names contain only ASCII alnum characters (e.g. mydb.t1) do not reproduce the bug -- the filesystem and system charset forms are byte-identical, so the strcmp divergence never arises.
[Suggested fix]
---------------
The bug has two equivalent root causes; either can be patched, but the consumer-side fix is safer because the producer side is shared by many recovery paths.
Producer side -- fil_op_replay_rename() in storage/innobase/fil/fil0fil.cc. The "rollback target file does NOT exist" branch (after the failed df.open_read_only()) reconstructs space->name via plain std::string slicing of the .ibd filepath, with no Fil_path::convert_to_filename_charset() / file_to_table() decoding. Fixing here would require decoding each '/'-separated component back to system charset before passing to fil_rename_tablespace(). Risky to change because this code is shared with crash-recovery replay where the on-disk path is the only source of truth.
Consumer side (recommended) -- dd_load_tablespace() in storage/innobase/dict/dict0dd.cc. In the "if (filepath != nullptr)" branch (the one that currently sets table->ibd_file_missing = true and returns), detect the divergence and reconcile the FIL cache name to the DD name before giving up. The primitive fil_space_update_name() already exists for exactly this purpose -- boot_tablespaces() uses it to reconcile a divergent FIL cache name against the DD on startup. After reconciliation, retry fil_space_exists_in_mem(); on success, the table opens normally, on failure, fall through to the current ibd_file_missing path.
In pseudocode:
if (filepath != nullptr) {
fil_space_t *space = fil_space_get(table->space);
if (space != nullptr && space->name != nullptr &&
strcmp(space->name, space_name) != 0) {
// Log the divergence as an operational signal.
// Reconcile cache name to DD name and retry.
fil_space_update_name(space, space_name);
if (is_already_opened()) {
// ...free filepath, return...
}
}
table->ibd_file_missing = true;
// ...existing cleanup...
}
The consumer-side fix is local, touches only the path that is currently broken, leaves crash-recovery replay untouched, and reuses an existing primitive. A full patch with regression tests covering all four name components (schema / table / partition / sub-partition) and both positive (fix engaged) and negative (fix bypassed -> bug reproduces) halves can be attached as a contribution.
[28 May 11:43]
Huaxiong Song
Fix of bug#120561 (*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.
Contribution: 0001-InnoDB-tablespace-marked-missing-after-rollback-of-c.patch (application/octet-stream, text), 20.72 KiB.
[29 May 4:41]
Huaxiong Song
Updated Fix (*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.
Contribution: 0001-InnoDB-tablespace-marked-missing-after-rollback-of-c.patch (application/octet-stream, text), 20.36 KiB.
Description: When a copy-style ALTER (ALGORITHM=COPY, e.g. ALTER TABLE t FORCE, ALGORITHM=COPY) fails into err_with_mdl after both mysql_rename_table() calls have already run, the rollback path in Log_DDL::post_ddl -> fil_op_replay_rename() enters its "rollback target file does NOT exist" branch (storage/innobase/fil/fil0fil.cc, around the name.erase(0, datadir_pos + 1) / name.append(...) section). That branch string-slices the .ibd filepath taken from the DDL log without performing any filename-to-tablename charset decoding, so the resulting fil_space_t::name ends up in filesystem charset (e.g. schema@002dx/t1). On the next access to the table, dd_load_tablespace() (storage/innobase/dict/dict0dd.cc) looks the space up under its system-charset name (schema-x/t1, decoded via convert_to_space() -> file_to_table()). fil_space_exists_in_mem()'s strcmp fails, so dd_load_tablespace() falls into the branch: if (filepath != nullptr) { /* If space id is already open with a different space name, then skip loading the space. ... */ table->ibd_file_missing = true; ... return; } The table is then reported as missing (ER_TABLESPACE_MISSING / "Tablespace is missing for table") and subsequent reads return zero rows, even though the .ibd file is fully intact on disk. Conditions to surface the bug ----------------------------- The bug is silent for pure ASCII-alnum names because their filesystem-charset and system-charset forms are byte-identical (no @NNNN encoding). It surfaces only when at least one of the following name components contains '-', non-ASCII characters, or any character that gets @NNNN-encoded on disk: - schema name (e.g. `mydb-x`) - table name (e.g. `t-name`) - partition name (e.g. `p-low`) - sub-partition name (e.g. `sp-a`) The bug is engine-agnostic with respect to the DDL caller -- any storage engine that flags HTON_SUPPORTS_ATOMIC_DDL and reaches err_with_mdl after both mysql_rename_table() calls in mysql_alter_table() triggers it. How to repeat: A debug-build reproduction is included below. The bug also surfaces in production builds whenever an atomic ALTER reaches `err_with_mdl` after both renames — for example via deadlock during `update_referencing_views_metadata()`, a foreign-key parent invalidation failure, or commit failure — but a deterministic non-debug repro requires precise concurrency, so the DBUG hook is added in `sql/sql_table.cc` purely to force the same code path that production failures take. 1. Build MySQL with `-DWITH_DEBUG=1`(mysql version is 9.7). 2. Add a one-line DBUG hook in `sql/sql_table.cc`, in `mysql_alter_table()`, immediately before the `update_referencing_views_metadata()` call inside the `if (!is_noop)` block (this is the only post-rename point where `err_with_mdl` is still reachable): ```cpp DBUG_EXECUTE_IF("force_alter_rollback_after_rename", { my_error(ER_LOCK_DEADLOCK, MYF(0)); goto err_with_mdl; }); ``` 3. Run: ```sql CREATE DATABASE `mydb-x`; USE `mydb-x`; CREATE TABLE t1 (id INT PRIMARY KEY, val VARCHAR(32)) ENGINE=InnoDB; INSERT INTO t1 VALUES (1, 'a'), (2, 'b'), (3, 'c'); FLUSH TABLES t1; SET SESSION debug = '+d,force_alter_rollback_after_rename'; ALTER TABLE t1 FORCE, ALGORITHM=COPY; -- ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction SET SESSION debug = '-d,force_alter_rollback_after_rename'; FLUSH TABLES t1; SELECT * FROM t1; -- ERROR 1812 (HY000): Tablespace is missing for table `mydb-x`.`t1`. ``` The `.ibd` file is fully present: ``` $ ls $datadir/mydb@002dx/ t1.ibd ``` Restarting the server makes the table readable again, which confirms the corruption is purely in the in-memory FIL cache, not on disk. 4. Equivalent reproductions for table-name, partition-name, and sub-partition-name variants by substituting: - table: `CREATE TABLE` `` `t-name` `` in any database - partition: `PARTITION` `` `p-low` `` `VALUES LESS THAN (50)` - sub-partition: `SUBPARTITION` `` `sp-a` `` In each case the `.ibd` on disk uses `@002d` encoding, while `dd_load_tablespace` looks up the system-charset form. 5. Tables/partitions whose names contain only ASCII alnum characters (e.g. `mydb.t1`) do **not** reproduce the bug — the filesystem and system charset forms are byte-identical, so the `strcmp` divergence never arises. Suggested fix: A debug-build reproduction is included below. The bug also surfaces in production builds whenever an atomic ALTER reaches err_with_mdl after both renames -- for example via deadlock during update_referencing_views_metadata(), a foreign-key parent invalidation failure, or commit failure -- but a deterministic non-debug repro requires precise concurrency, so the DBUG hook is added in sql/sql_table.cc purely to force the same code path that production failures take. 1. Build MySQL with -DWITH_DEBUG=1. 2. Add a DBUG hook in sql/sql_table.cc, in mysql_alter_table(), inside the "if (!is_noop)" block immediately before the update_referencing_views_metadata() call (this is the only post-rename point where err_with_mdl is still reachable). The hook is added between the existing uncommitted_tables setup and the existing update_referencing_views_metadata() call: if (write_bin_log(thd, true, thd->query().str, thd->query().length, atomic_replace && !is_noop)) goto err_with_mdl; if (!is_noop) { Uncommitted_tables_guard uncommitted_tables(thd); uncommitted_tables.add_table(table_list); + /* >>> ADDED: force copy-style ALTER past both mysql_rename_table + calls to fail into err_with_mdl, triggering the rollback branch + in fil_op_replay_rename. <<< */ + DBUG_EXECUTE_IF("force_alter_rollback_after_rename", { + my_error(ER_LOCK_DEADLOCK, MYF(0)); + goto err_with_mdl; + }); if (update_referencing_views_metadata(thd, table_list, new_db, new_name, !atomic_replace, &uncommitted_tables)) goto err_with_mdl; if (alter_ctx.is_table_renamed()) tdc_remove_table(thd, TDC_RT_REMOVE_ALL, alter_ctx.new_db, alter_ctx.new_name, false); } 3. Run: CREATE DATABASE `mydb-x`; USE `mydb-x`; CREATE TABLE t1 (id INT PRIMARY KEY, val VARCHAR(32)) ENGINE=InnoDB; INSERT INTO t1 VALUES (1, 'a'), (2, 'b'), (3, 'c'); FLUSH TABLES t1; SET SESSION debug = '+d,force_alter_rollback_after_rename'; ALTER TABLE t1 FORCE, ALGORITHM=COPY; -- ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction SET SESSION debug = '-d,force_alter_rollback_after_rename'; FLUSH TABLES t1; SELECT * FROM t1; -- ERROR 1812 (HY000): Tablespace is missing for table `mydb-x`.`t1`. The .ibd file is fully present: $ ls $datadir/mydb@002dx/ t1.ibd Restarting the server makes the table readable again, which confirms the corruption is purely in the in-memory FIL cache, not on disk. 4. Equivalent reproductions for table-name, partition-name, and sub-partition-name variants by substituting: - table: CREATE TABLE `t-name` in any database - partition: PARTITION `p-low` VALUES LESS THAN (50) - sub-partition: SUBPARTITION `sp-a` In each case the .ibd on disk uses @002d encoding, while dd_load_tablespace looks up the system-charset form. 5. Tables/partitions whose names contain only ASCII alnum characters (e.g. mydb.t1) do not reproduce the bug -- the filesystem and system charset forms are byte-identical, so the strcmp divergence never arises.