MySQL Bugs: #119978: In-place upgrade of Mysql 5.7.x/8.0.19 to revision after 8.0.23 fails due to undo

Bug #119978	In-place upgrade of Mysql 5.7.x/8.0.19 to revision after 8.0.23 fails due to undo_00x tablespace ID
Submitted:	3 Mar 16:41	Modified:	8 Apr 9:16
Reporter:	Yordan Nanchev	Email Updates:
Status:	Open	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S3 (Non-critical)
Version:	8.0	OS:	SUSE (SLES 15 SP6)
Assigned to:	ADITYA ANANTHAPADMANABHA	CPU Architecture:	x86
Tags:	Upgrade failed

Description:
Mysql 5.7.44 can be upgraded in-place to 8.0 up to 8.0.19
Upgrade of the same database from version 5.7.44 to 8.0.45, 
or from 8.0.19 to 8.0.45, fails because undo_001/undo_002 cannot be created - tablespace ID used for them are from existing tables.

Errors for upgrade 5.7.44 -> 8.0.45
2026-03-02T15:33:30.179729Z 1 [Note] [MY-012905] [InnoDB] Cannot create ./undo_001 because ./db/tbl.ibd already uses Space ID=4255581912! Did you change innodb_undo_directory?
2026-03-02T15:33:30.179767Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Invalid Filename.
2026-03-02T15:33:30.364701Z 1 [ERROR] [MY-011013] [Server] Failed to initialize DD Storage Engine.
2026-03-02T15:33:30.365002Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2026-03-02T15:33:30.365051Z 0 [ERROR] [MY-010119] [Server] Aborting

Errors for upgrade 8.0.19 -> 8.0.45
2026-03-02T19:24:21.029710Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_001'.
2026-03-02T19:24:21.030658Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_002'.
2026-03-02T19:24:21.031162Z 1 [Note] [MY-013261] [InnoDB] Cannot use ./db/tbl.ibd as an undo tablespace because it does not end with '.ibu'.
2026-03-02T19:24:21.031180Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Invalid Filename.
2026-03-02T19:24:21.495525Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2026-03-02T19:24:21.495864Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2026-03-02T19:24:21.495925Z 0 [ERROR] [MY-010119] [Server] Aborting

Upgrade from 5.7 works also for revisions before 8.0.19
Upgarde from 8.0.19 fails also for revisions before 8.0.45 - like 8.0.43/8.0.21
Server is SUSE Linux Enterprise Server 15 SP6

How to repeat:
application that uses the database heavily uses temporary tables -  creates/destroys a lot of them daily.

Suggested fix:
check if tablespace ID is in use before assigning it to undo files during the upgrade

1. Before 8.0.24, we didn’t have any dedicated reserved space IDs for the undo tablespace. So, it is theoretically possible that a table could be allocated a tablespace ID (4255581912).

2. I can see the errors reported on the bug page after I manually allocate this ID to a permanent table.

3. The question is: how did we end up here? Although the customer says they are using many temporary tables, I noticed that this didn’t actually increase the space ID counter for permanent tables.

4. One possibility is that they are doing a lot of create/drop operations on permanent tables, or running TRUNCATE operations, which would bump up the space ID counter.

5. On the other hand, it is virtually impossible to reach this number. Even if we created a table every second for 10 years, we would not reach it.

6. So, I suspect there may be a bug somewhere that corrupts the space ID counter.

7. Is it possible to determine what operations they are performing so that we can reproduce the issue?
   (i) Are they doing a lot of create/drop operations on permanent tables?  
   (ii) Are they running TRUNCATE frequently from multiple threads ?

8. To solve the customer’s upgrade problem, it would be better to ask the customer to perform a logical upgrade (using `mysqldump`) instead of an in-place upgrade:  
   https://dev.mysql.com/doc/refman/8.0/en/copying-databases.html

to answer p. 7:
customer application creates and drops a lot of permanent tables (ibd).
For reference here are highest table_id for oldest databases:
select max(FILE_ID) from information_schema.files
 4255814568 
 3340577725 
 3022178494 
 2952805509 
I have upgraded the problem one with mysqldump/restore, it seems others will not be affected.

1. I would suggest to use temporary tables ,if they are using lots of drop and 
   create instead of permanent tables. These are faster (since no redo logs) and 
   will not cause any problems during upgrade.

2. Please create a new permanent table after upgrade and check what space id is being allocated to it . If it is still a large number, then this problem can come again.
So it would be better to dump and restore all the tables in any 8.0 version. If this was a 5.7 issue only (allocating large space id's) then dump and restore will eliminate this in higher versions

3. I am still not sure how to reproduce the issue. Just creating and dropping may not reporoduce this problem. As i mentioned, to reach this number is virtually impossible. Do they have a test scenario which they can run from a newly inititiliazed mysqld server that reproduces this?

- after dump/restore issue is resolving
- there are no test case, and I suspect this were to an application error a few years back. Currently no more than 200 tables are created/dropped per day

Can we close this bug as not reproducible ?

yes

> [2 Apr 8:07] ADITYA ANANTHAPADMANABHA
>
> Can we close this bug as not reproducible ?

> [2 Apr 14:19] Yordan Nanchev
>
> yes

On my side, I am not sure it is ok to close as not reproducible.

> [20 Mar 5:48] ADITYA ANANTHAPADMANABHA
>
> 2. I can see the errors reported on the bug page after I manually allocate this ID to a permanent table.

Above makes me think it is reproducible.  The fact we do not know how how we got here does not mean it is not reproducible.  A repro might be tedious, but I think we all agree there is an unwanted behavior here.

Now, is it important to fix this, I do not think so.  We know of a workaround (mysqldump), and there might even be others (getting rid of this Space ID with EXPORT / IMPORT TABLESPACE, or CREATE TABLE LIKE and INSERT SELECT).

The right status for me would be "Will not fix".

Alternatively, the MySQL Upgrade Check Utility (probably wrong name) should probably warn in such case, explaining how to fix the situation.

And it probably also affects 8.4, as from what I understand, upgrading from before 8.0.19 to 8.4 should be possible.  And this supports the case for updating the Upgrade Check Utility in 8.4.  Especially that such failed upgrade might leave users in a very bad situation (database unusable ?).

Adding the check in mysql 8/8.4 "upgrade checker" if tablespace IDs are in use would be very helpful indeed (showing which tables are affected)