Bug #120561 tablespace marked missing after rollback of copy-style ALTER

Description:
When a copy-style ALTER (ALGORITHM=COPY, e.g. ALTER TABLE t FORCE, ALGORITHM=COPY) fails into err_with_mdl after both mysql_rename_table() calls have already run, the rollback path in Log_DDL::post_ddl -> fil_op_replay_rename() enters its "rollback target file does NOT exist" branch (storage/innobase/fil/fil0fil.cc, around the name.erase(0, datadir_pos + 1) / name.append(...) section). That branch string-slices the .ibd filepath taken from the DDL log without performing any filename-to-tablename charset decoding, so the resulting fil_space_t::name ends up in filesystem charset (e.g. schema@002dx/t1).

On the next access to the table, dd_load_tablespace() (storage/innobase/dict/dict0dd.cc) looks the space up under its system-charset name (schema-x/t1, decoded via convert_to_space() -> file_to_table()). fil_space_exists_in_mem()'s strcmp fails, so dd_load_tablespace() falls into the branch:

    if (filepath != nullptr) {
        /* If space id is already open with a different space name, then skip
        loading the space. ... */
        table->ibd_file_missing = true;
        ...
        return;
    }

The table is then reported as missing (ER_TABLESPACE_MISSING / "Tablespace is missing for table") and subsequent reads return zero rows, even though the .ibd file is fully intact on disk.

Conditions to surface the bug
-----------------------------

The bug is silent for pure ASCII-alnum names because their filesystem-charset and system-charset forms are byte-identical (no @NNNN encoding). It surfaces only when at least one of the following name components contains '-', non-ASCII characters, or any character that gets @NNNN-encoded on disk:

  - schema name (e.g. `mydb-x`)
  - table name (e.g. `t-name`)
  - partition name (e.g. `p-low`)
  - sub-partition name (e.g. `sp-a`)

The bug is engine-agnostic with respect to the DDL caller -- any storage engine that flags HTON_SUPPORTS_ATOMIC_DDL and reaches err_with_mdl after both mysql_rename_table() calls in mysql_alter_table() triggers it.

How to repeat:
A debug-build reproduction is included below. The bug also surfaces in production builds whenever an atomic ALTER reaches `err_with_mdl` after both renames — for example via deadlock during `update_referencing_views_metadata()`, a foreign-key parent invalidation failure, or commit failure — but a deterministic non-debug repro requires precise concurrency, so the DBUG hook is added in `sql/sql_table.cc` purely to force the same code path that production failures take.

1. Build MySQL with `-DWITH_DEBUG=1`(mysql version is 9.7).

2. Add a one-line DBUG hook in `sql/sql_table.cc`, in `mysql_alter_table()`, immediately before the `update_referencing_views_metadata()` call inside the `if (!is_noop)` block (this is the only post-rename point where `err_with_mdl` is still reachable):

   ```cpp
   DBUG_EXECUTE_IF("force_alter_rollback_after_rename", {
     my_error(ER_LOCK_DEADLOCK, MYF(0));
     goto err_with_mdl;
   });
   ```

3. Run:

   ```sql
   CREATE DATABASE `mydb-x`;
   USE `mydb-x`;
   CREATE TABLE t1 (id INT PRIMARY KEY, val VARCHAR(32)) ENGINE=InnoDB;
   INSERT INTO t1 VALUES (1, 'a'), (2, 'b'), (3, 'c');
   FLUSH TABLES t1;

   SET SESSION debug = '+d,force_alter_rollback_after_rename';
   ALTER TABLE t1 FORCE, ALGORITHM=COPY;
   -- ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
   SET SESSION debug = '-d,force_alter_rollback_after_rename';

   FLUSH TABLES t1;
   SELECT * FROM t1;
   -- ERROR 1812 (HY000): Tablespace is missing for table `mydb-x`.`t1`.
   ```

   The `.ibd` file is fully present:

   ```
   $ ls $datadir/mydb@002dx/
   t1.ibd
   ```

   Restarting the server makes the table readable again, which confirms the corruption is purely in the in-memory FIL cache, not on disk.

4. Equivalent reproductions for table-name, partition-name, and sub-partition-name variants by substituting:

   - table: `CREATE TABLE` `` `t-name` `` in any database
   - partition: `PARTITION` `` `p-low` `` `VALUES LESS THAN (50)`
   - sub-partition: `SUBPARTITION` `` `sp-a` ``

   In each case the `.ibd` on disk uses `@002d` encoding, while `dd_load_tablespace` looks up the system-charset form.

5. Tables/partitions whose names contain only ASCII alnum characters (e.g. `mydb.t1`) do **not** reproduce the bug — the filesystem and system charset forms are byte-identical, so the `strcmp` divergence never arises.

Suggested fix:
A debug-build reproduction is included below. The bug also surfaces in production builds whenever an atomic ALTER reaches err_with_mdl after both renames -- for example via deadlock during update_referencing_views_metadata(), a foreign-key parent invalidation failure, or commit failure -- but a deterministic non-debug repro requires precise concurrency, so the DBUG hook is added in sql/sql_table.cc purely to force the same code path that production failures take.

1. Build MySQL with -DWITH_DEBUG=1.

2. Add a DBUG hook in sql/sql_table.cc, in mysql_alter_table(), inside the
   "if (!is_noop)" block immediately before the update_referencing_views_metadata()
   call (this is the only post-rename point where err_with_mdl is still
   reachable). The hook is added between the existing uncommitted_tables setup
   and the existing update_referencing_views_metadata() call:

       if (write_bin_log(thd, true, thd->query().str, thd->query().length,
                         atomic_replace && !is_noop))
         goto err_with_mdl;

       if (!is_noop) {
         Uncommitted_tables_guard uncommitted_tables(thd);

         uncommitted_tables.add_table(table_list);

         + /* >>> ADDED: force copy-style ALTER past both mysql_rename_table
         +   calls to fail into err_with_mdl, triggering the rollback branch
         +   in fil_op_replay_rename. <<< */
         + DBUG_EXECUTE_IF("force_alter_rollback_after_rename", {
         +  my_error(ER_LOCK_DEADLOCK, MYF(0));
         +  goto err_with_mdl;
         + });

         if (update_referencing_views_metadata(thd, table_list, new_db, new_name,
                                               !atomic_replace, &uncommitted_tables))
           goto err_with_mdl;

         if (alter_ctx.is_table_renamed())
           tdc_remove_table(thd, TDC_RT_REMOVE_ALL, alter_ctx.new_db,
                            alter_ctx.new_name, false);
       }

3. Run:

       CREATE DATABASE `mydb-x`;
       USE `mydb-x`;
       CREATE TABLE t1 (id INT PRIMARY KEY, val VARCHAR(32)) ENGINE=InnoDB;
       INSERT INTO t1 VALUES (1, 'a'), (2, 'b'), (3, 'c');
       FLUSH TABLES t1;

       SET SESSION debug = '+d,force_alter_rollback_after_rename';
       ALTER TABLE t1 FORCE, ALGORITHM=COPY;
       -- ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
       SET SESSION debug = '-d,force_alter_rollback_after_rename';

       FLUSH TABLES t1;
       SELECT * FROM t1;
       -- ERROR 1812 (HY000): Tablespace is missing for table `mydb-x`.`t1`.

   The .ibd file is fully present:

       $ ls $datadir/mydb@002dx/
       t1.ibd

   Restarting the server makes the table readable again, which confirms the corruption is purely in the in-memory FIL cache, not on disk.

4. Equivalent reproductions for table-name, partition-name, and sub-partition-name variants by substituting:

   - table:         CREATE TABLE `t-name` in any database
   - partition:     PARTITION `p-low` VALUES LESS THAN (50)
   - sub-partition: SUBPARTITION `sp-a`

   In each case the .ibd on disk uses @002d encoding, while dd_load_tablespace looks up the system-charset form.

5. Tables/partitions whose names contain only ASCII alnum characters (e.g. mydb.t1) do not reproduce the bug -- the filesystem and system charset forms are byte-identical, so the strcmp divergence never arises.

Sorry, the content under “Suggested fix” overlaps with “How to Repeat.” Let me reorganize it and send an updated version.

=========================================================================================================================

[How to repeat]
---------------

A debug-build reproduction is included below. The bug also surfaces in production builds whenever an atomic ALTER reaches err_with_mdl after both renames -- for example via deadlock during update_referencing_views_metadata(), a foreign-key parent invalidation failure, or commit failure -- but a deterministic non-debug repro requires precise concurrency, so the DBUG hook is added in sql/sql_table.cc purely to force the same code path that production failures take.

1. Build MySQL with -DWITH_DEBUG=1.

2. Add a DBUG hook in sql/sql_table.cc, in mysql_alter_table(), inside the
   "if (!is_noop)" block immediately before the update_referencing_views_metadata()
   call (this is the only post-rename point where err_with_mdl is still
   reachable). The hook is added between the existing uncommitted_tables setup
   and the existing update_referencing_views_metadata() call:

       if (write_bin_log(thd, true, thd->query().str, thd->query().length,
                         atomic_replace && !is_noop))
         goto err_with_mdl;

       if (!is_noop) {
         Uncommitted_tables_guard uncommitted_tables(thd);

         uncommitted_tables.add_table(table_list);

         /* >>> ADDED: force copy-style ALTER past both mysql_rename_table
            calls to fail into err_with_mdl, triggering the rollback branch
            in fil_op_replay_rename. <<< */
         DBUG_EXECUTE_IF("force_alter_rollback_after_rename", {
           my_error(ER_LOCK_DEADLOCK, MYF(0));
           goto err_with_mdl;
         });

         if (update_referencing_views_metadata(thd, table_list, new_db, new_name,
                                               !atomic_replace, &uncommitted_tables))
           goto err_with_mdl;

         if (alter_ctx.is_table_renamed())
           tdc_remove_table(thd, TDC_RT_REMOVE_ALL, alter_ctx.new_db,
                            alter_ctx.new_name, false);
       }

3. Run:

       CREATE DATABASE `mydb-x`;
       USE `mydb-x`;
       CREATE TABLE t1 (id INT PRIMARY KEY, val VARCHAR(32)) ENGINE=InnoDB;
       INSERT INTO t1 VALUES (1, 'a'), (2, 'b'), (3, 'c');
       FLUSH TABLES t1;

       SET SESSION debug = '+d,force_alter_rollback_after_rename';
       ALTER TABLE t1 FORCE, ALGORITHM=COPY;
       -- ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
       SET SESSION debug = '-d,force_alter_rollback_after_rename';

       FLUSH TABLES t1;
       SELECT * FROM t1;
       -- ERROR 1812 (HY000): Tablespace is missing for table `mydb-x`.`t1`.

   The .ibd file is fully present:

       $ ls $datadir/mydb@002dx/
       t1.ibd

   Restarting the server makes the table readable again, which confirms the corruption is purely in the in-memory FIL cache, not on disk.

4. Equivalent reproductions for table-name, partition-name, and sub-partition-name variants by substituting:

   - table:         CREATE TABLE `t-name` in any database
   - partition:     PARTITION `p-low` VALUES LESS THAN (50)
   - sub-partition: SUBPARTITION `sp-a`

   In each case the .ibd on disk uses @002d encoding, while dd_load_tablespace looks up the system-charset form.

5. Tables/partitions whose names contain only ASCII alnum characters (e.g. mydb.t1) do not reproduce the bug -- the filesystem and system charset forms are byte-identical, so the strcmp divergence never arises.

[Suggested fix]
---------------

The bug has two equivalent root causes; either can be patched, but the consumer-side fix is safer because the producer side is shared by many recovery paths.

Producer side -- fil_op_replay_rename() in storage/innobase/fil/fil0fil.cc. The "rollback target file does NOT exist" branch (after the failed df.open_read_only()) reconstructs space->name via plain std::string slicing of the .ibd filepath, with no Fil_path::convert_to_filename_charset() / file_to_table() decoding. Fixing here would require decoding each '/'-separated component back to system charset before passing to fil_rename_tablespace(). Risky to change because this code is shared with crash-recovery replay where the on-disk path is the only source of truth.

Consumer side (recommended) -- dd_load_tablespace() in storage/innobase/dict/dict0dd.cc. In the "if (filepath != nullptr)" branch (the one that currently sets table->ibd_file_missing = true and returns), detect the divergence and reconcile the FIL cache name to the DD name before giving up. The primitive fil_space_update_name() already exists for exactly this purpose -- boot_tablespaces() uses it to reconcile a divergent FIL cache name against the DD on startup. After reconciliation, retry fil_space_exists_in_mem(); on success, the table opens normally, on failure, fall through to the current ibd_file_missing path.

In pseudocode:

    if (filepath != nullptr) {
        fil_space_t *space = fil_space_get(table->space);
        if (space != nullptr && space->name != nullptr &&
            strcmp(space->name, space_name) != 0) {
            // Log the divergence as an operational signal.
            // Reconcile cache name to DD name and retry.
            fil_space_update_name(space, space_name);
            if (is_already_opened()) {
                // ...free filepath, return...
            }
        }
        table->ibd_file_missing = true;
        // ...existing cleanup...
    }

The consumer-side fix is local, touches only the path that is currently broken, leaves crash-recovery replay untouched, and reuses an existing primitive. A full patch with regression tests covering all four name components (schema / table / partition / sub-partition) and both positive (fix engaged) and negative (fix bypassed -> bug reproduces) halves can be attached as a contribution.

Fix of bug#120561

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: 0001-InnoDB-tablespace-marked-missing-after-rollback-of-c.patch (application/octet-stream, text), 20.72 KiB.

Updated Fix

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: 0001-InnoDB-tablespace-marked-missing-after-rollback-of-c.patch (application/octet-stream, text), 20.36 KiB.