Description:
I'm trying to upgrade from 8.0.40 to 8.0.42 with one of the schema having ~1 million tables.
mysql> SELECT table_schema, COUNT(*) as table_count FROM information_schema.tables GROUP BY table_schema;
+--------------------+-------------+
| TABLE_SCHEMA | table_count |
+--------------------+-------------+
| information_schema | 81 |
| mysql | 47 |
| performance_schema | 111 |
| schema1 | 1042000 |
| schema2 | 15000 |
| sys | 101 |
+--------------------+-------------+
6 rows in set (4.34 sec)
Running upgrade on a server with 64GB of RAM failed and mysqld was killed by OOM killer during the upgrade.
$ free -h
total used free shared buff/cache available
Mem: 62G 60G 623M 744K 1.1G 756M
Swap: 15G 335M 15G
> The upgrade successfully finished on server with 128GB RAM and mysqld consumed ~100GB memory at peak and upgrade took ~2hr17mins
2025-09-17T18:04:07.135304Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2025-09-17T18:04:07.161603Z 1 [Note] [MY-013327] [Server] MySQL server upgrading from version '80040' to '80042'.
2025-09-17T18:04:07.585359Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2025-09-17T18:04:07.586277Z 1 [Note] [MY-012356] [InnoDB] Scanned 13 tablespaces. Validated 13.
2025-09-17T20:21:27.371983Z 1 [Note] [MY-010006] [Server] Using data dictionary with version '80023'.
2025-09-17T20:21:30.371434Z 4 [System] [MY-013381] [Server] Server upgrade from '80040' to '80042' started.
2025-09-17T20:21:30.419538Z 4 [Note] [MY-013386] [Server] Running queries to upgrade MySQL server.
2025-09-17T20:21:41.562814Z 4 [Note] [MY-013387] [Server] Upgrading system table data.
2025-09-17T20:21:42.310122Z 4 [Note] [MY-013385] [Server] Upgrading the sys schema.
2025-09-17T20:21:48.199753Z 4 [Note] [MY-013400] [Server] Upgrade of help tables started.
2025-09-17T20:21:48.401013Z 4 [Note] [MY-013400] [Server] Upgrade of help tables completed.
2025-09-17T20:21:48.403309Z 4 [Note] [MY-013394] [Server] Checking 'mysql' schema.
2025-09-17T20:21:49.289799Z 4 [Note] [MY-013394] [Server] Checking 'sys' schema.
2025-09-17T20:21:49.673758Z 4 [System] [MY-013381] [Server] Server upgrade from '80040' to '80042' completed.
> Upgrading to 8.0.41 didn't consume as much memory and upgrade finished in just ~9mins
2025-09-17T21:44:20.081295Z 1 [Note] [MY-012976] [InnoDB] 8.0.41 started; log sequence number 39330580770
2025-09-17T21:44:20.082075Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2025-09-17T21:44:20.108979Z 1 [Note] [MY-013327] [Server] MySQL server upgrading from version '80040' to '80041'.
2025-09-17T21:44:20.515536Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2025-09-17T21:44:20.516508Z 1 [Note] [MY-012356] [InnoDB] Scanned 13 tablespaces. Validated 13.
2025-09-17T21:53:06.055438Z 1 [Note] [MY-010006] [Server] Using data dictionary with version '80023'.
2025-09-17T21:53:06.080065Z 4 [System] [MY-013381] [Server] Server upgrade from '80040' to '80041' started.
2025-09-17T21:53:06.080899Z 4 [Note] [MY-013386] [Server] Running queries to upgrade MySQL server.
2025-09-17T21:53:09.011848Z 4 [Note] [MY-013387] [Server] Upgrading system table data.
2025-09-17T21:53:09.685842Z 4 [Note] [MY-013385] [Server] Upgrading the sys schema.
2025-09-17T21:53:10.498516Z 4 [Note] [MY-013400] [Server] Upgrade of help tables started.
2025-09-17T21:53:10.632588Z 4 [Note] [MY-013400] [Server] Upgrade of help tables completed.
2025-09-17T21:53:10.632677Z 4 [Note] [MY-013394] [Server] Checking 'mysql' schema.
2025-09-17T21:53:10.774560Z 4 [Note] [MY-013394] [Server] Checking 'sys' schema.
2025-09-17T21:53:10.791542Z 4 [System] [MY-013381] [Server] Server upgrade from '80040' to '80041' completed.
Based on the above testing, I suspect this behavior is a consequence of the change in 8.0.42 which added a new check with function check_table_funs as part of function do_server_upgrade_checks during upgrade.
Code:
https://github.com/mysql/mysql-server/blob/mysql-cluster-8.0.42/sql/dd/impl/upgrade/server...
Release notes:
https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-42.html
----------------------------------
pstack for thread running upgrade: (frame #10)
$ pstack 14716
Thread 1 (Thread 0x4000494ff550 (LWP 14716)):
#0 0x000000000136aafc in void* my_internal_malloc<&(redirecting_allocator(unsigned long, int))>(unsigned int, unsigned long, int) [clone .lto_priv.0] ()
#1 0x00000000009abff8 in open_table(THD*, Table_ref*, Open_table_context*) ()
#2 0x00000000009ad338 in open_tables(THD*, Table_ref**, unsigned int*, unsigned int, Prelocking_strategy*) ()
#3 0x000000000137ac88 in dd::Open_dictionary_tables_ctx::open_tables() ()
#4 0x000000000129cdcc in bool dd::cache::Storage_adapter::get<dd::Item_name_key, dd::Abstract_table>(THD*, dd::Item_name_key const&, enum_tx_isolation, bool, dd::Abstract_table const**) ()
#5 0x000000000129cfcc in bool dd::cache::Shared_dictionary_cache::get<dd::Item_name_key, dd::Abstract_table>(THD*, dd::Item_name_key const&, dd::cache::Cache_element<dd::Abstract_table>**) ()
#6 0x0000000001b1e698 in bool dd::cache::Dictionary_client::acquire<dd::Item_name_key, dd::Abstract_table>(dd::Item_name_key const&, dd::Abstract_table const**, bool*, bool*) [clone .constprop.0] ()
#7 0x0000000001252fb8 in bool dd::cache::Dictionary_client::acquire<dd::Abstract_table>(std::__cxx11::basic_string<char, std::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, dd::Abstract_table const**) ()
#8 0x00000000009a96b8 in get_table_share(THD*, char const*, char const*, char const*, unsigned long, bool, bool) ()
#9 0x00000000009abb7c in open_table(THD*, Table_ref*, Open_table_context*) ()
#10 0x000000000135bd04 in dd::upgrade::(anonymous namespace)::check_table_funs(THD*, std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&, dd::upgrade::Upgrade_error_counter*)::{lambda(std::unique_ptr<dd::Table, std::default_delete<dd::Table> >&)#1}::operator()(std::unique_ptr<dd::Table, std::default_delete<dd::Table> >&) const ()
#11 0x00000000012725d4 in bool dd::cache::Dictionary_client::foreach<dd::Table>(dd::Object_key const*, std::function<bool (std::unique_ptr<dd::Table, std::default_delete<dd::Table> >&)> const&) const ()
#12 0x0000000001352920 in std::_Function_handler<bool (std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&), dd::upgrade::do_server_upgrade_checks(THD*)::{lambda(std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&)#1}>::_M_invoke(std::_Any_data const&, std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&) ()
#13 0x0000000001b26da4 in bool dd::cache::Dictionary_client::foreach<dd::Schema>(dd::Object_key const*, std::function<bool (std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&)> const&) const [clone .constprop.0] ()
#14 0x0000000001359294 in dd::upgrade::do_server_upgrade_checks(THD*) ()
#15 0x0000000001254fa4 in dd::bootstrap::restart(THD*) ()
#16 0x000000000137ab74 in dd::upgrade_57::restart_dictionary(THD*) ()
#17 0x000000000138b718 in dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*) ()
#18 0x0000000000bcafe4 in handle_bootstrap ()
#19 0x00000000016d4d40 in pfs_spawn_thread ()
#20 0x0000400033bec230 in start_thread () from target:/lib64/libpthread.so.0
#21 0x0000400033de77dc in thread_start () from target:/lib64/libc.so.6
How to repeat:
1. Create a MySQL server 8.0.40
2. Create 1 Million tables in a schema.
3. Upgrade to 8.0.41 this should be quick.
4. Try to upgrade to 8.0.42 - issue will be reproduced.
© 2025, Oracle Corporation and/or its affiliates
Description: I'm trying to upgrade from 8.0.40 to 8.0.42 with one of the schema having ~1 million tables. mysql> SELECT table_schema, COUNT(*) as table_count FROM information_schema.tables GROUP BY table_schema; +--------------------+-------------+ | TABLE_SCHEMA | table_count | +--------------------+-------------+ | information_schema | 81 | | mysql | 47 | | performance_schema | 111 | | schema1 | 1042000 | | schema2 | 15000 | | sys | 101 | +--------------------+-------------+ 6 rows in set (4.34 sec) Running upgrade on a server with 64GB of RAM failed and mysqld was killed by OOM killer during the upgrade. $ free -h total used free shared buff/cache available Mem: 62G 60G 623M 744K 1.1G 756M Swap: 15G 335M 15G > The upgrade successfully finished on server with 128GB RAM and mysqld consumed ~100GB memory at peak and upgrade took ~2hr17mins 2025-09-17T18:04:07.135304Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2025-09-17T18:04:07.161603Z 1 [Note] [MY-013327] [Server] MySQL server upgrading from version '80040' to '80042'. 2025-09-17T18:04:07.585359Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files 2025-09-17T18:04:07.586277Z 1 [Note] [MY-012356] [InnoDB] Scanned 13 tablespaces. Validated 13. 2025-09-17T20:21:27.371983Z 1 [Note] [MY-010006] [Server] Using data dictionary with version '80023'. 2025-09-17T20:21:30.371434Z 4 [System] [MY-013381] [Server] Server upgrade from '80040' to '80042' started. 2025-09-17T20:21:30.419538Z 4 [Note] [MY-013386] [Server] Running queries to upgrade MySQL server. 2025-09-17T20:21:41.562814Z 4 [Note] [MY-013387] [Server] Upgrading system table data. 2025-09-17T20:21:42.310122Z 4 [Note] [MY-013385] [Server] Upgrading the sys schema. 2025-09-17T20:21:48.199753Z 4 [Note] [MY-013400] [Server] Upgrade of help tables started. 2025-09-17T20:21:48.401013Z 4 [Note] [MY-013400] [Server] Upgrade of help tables completed. 2025-09-17T20:21:48.403309Z 4 [Note] [MY-013394] [Server] Checking 'mysql' schema. 2025-09-17T20:21:49.289799Z 4 [Note] [MY-013394] [Server] Checking 'sys' schema. 2025-09-17T20:21:49.673758Z 4 [System] [MY-013381] [Server] Server upgrade from '80040' to '80042' completed. > Upgrading to 8.0.41 didn't consume as much memory and upgrade finished in just ~9mins 2025-09-17T21:44:20.081295Z 1 [Note] [MY-012976] [InnoDB] 8.0.41 started; log sequence number 39330580770 2025-09-17T21:44:20.082075Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2025-09-17T21:44:20.108979Z 1 [Note] [MY-013327] [Server] MySQL server upgrading from version '80040' to '80041'. 2025-09-17T21:44:20.515536Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files 2025-09-17T21:44:20.516508Z 1 [Note] [MY-012356] [InnoDB] Scanned 13 tablespaces. Validated 13. 2025-09-17T21:53:06.055438Z 1 [Note] [MY-010006] [Server] Using data dictionary with version '80023'. 2025-09-17T21:53:06.080065Z 4 [System] [MY-013381] [Server] Server upgrade from '80040' to '80041' started. 2025-09-17T21:53:06.080899Z 4 [Note] [MY-013386] [Server] Running queries to upgrade MySQL server. 2025-09-17T21:53:09.011848Z 4 [Note] [MY-013387] [Server] Upgrading system table data. 2025-09-17T21:53:09.685842Z 4 [Note] [MY-013385] [Server] Upgrading the sys schema. 2025-09-17T21:53:10.498516Z 4 [Note] [MY-013400] [Server] Upgrade of help tables started. 2025-09-17T21:53:10.632588Z 4 [Note] [MY-013400] [Server] Upgrade of help tables completed. 2025-09-17T21:53:10.632677Z 4 [Note] [MY-013394] [Server] Checking 'mysql' schema. 2025-09-17T21:53:10.774560Z 4 [Note] [MY-013394] [Server] Checking 'sys' schema. 2025-09-17T21:53:10.791542Z 4 [System] [MY-013381] [Server] Server upgrade from '80040' to '80041' completed. Based on the above testing, I suspect this behavior is a consequence of the change in 8.0.42 which added a new check with function check_table_funs as part of function do_server_upgrade_checks during upgrade. Code: https://github.com/mysql/mysql-server/blob/mysql-cluster-8.0.42/sql/dd/impl/upgrade/server... Release notes: https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-42.html ---------------------------------- pstack for thread running upgrade: (frame #10) $ pstack 14716 Thread 1 (Thread 0x4000494ff550 (LWP 14716)): #0 0x000000000136aafc in void* my_internal_malloc<&(redirecting_allocator(unsigned long, int))>(unsigned int, unsigned long, int) [clone .lto_priv.0] () #1 0x00000000009abff8 in open_table(THD*, Table_ref*, Open_table_context*) () #2 0x00000000009ad338 in open_tables(THD*, Table_ref**, unsigned int*, unsigned int, Prelocking_strategy*) () #3 0x000000000137ac88 in dd::Open_dictionary_tables_ctx::open_tables() () #4 0x000000000129cdcc in bool dd::cache::Storage_adapter::get<dd::Item_name_key, dd::Abstract_table>(THD*, dd::Item_name_key const&, enum_tx_isolation, bool, dd::Abstract_table const**) () #5 0x000000000129cfcc in bool dd::cache::Shared_dictionary_cache::get<dd::Item_name_key, dd::Abstract_table>(THD*, dd::Item_name_key const&, dd::cache::Cache_element<dd::Abstract_table>**) () #6 0x0000000001b1e698 in bool dd::cache::Dictionary_client::acquire<dd::Item_name_key, dd::Abstract_table>(dd::Item_name_key const&, dd::Abstract_table const**, bool*, bool*) [clone .constprop.0] () #7 0x0000000001252fb8 in bool dd::cache::Dictionary_client::acquire<dd::Abstract_table>(std::__cxx11::basic_string<char, std::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, dd::Abstract_table const**) () #8 0x00000000009a96b8 in get_table_share(THD*, char const*, char const*, char const*, unsigned long, bool, bool) () #9 0x00000000009abb7c in open_table(THD*, Table_ref*, Open_table_context*) () #10 0x000000000135bd04 in dd::upgrade::(anonymous namespace)::check_table_funs(THD*, std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&, dd::upgrade::Upgrade_error_counter*)::{lambda(std::unique_ptr<dd::Table, std::default_delete<dd::Table> >&)#1}::operator()(std::unique_ptr<dd::Table, std::default_delete<dd::Table> >&) const () #11 0x00000000012725d4 in bool dd::cache::Dictionary_client::foreach<dd::Table>(dd::Object_key const*, std::function<bool (std::unique_ptr<dd::Table, std::default_delete<dd::Table> >&)> const&) const () #12 0x0000000001352920 in std::_Function_handler<bool (std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&), dd::upgrade::do_server_upgrade_checks(THD*)::{lambda(std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&)#1}>::_M_invoke(std::_Any_data const&, std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&) () #13 0x0000000001b26da4 in bool dd::cache::Dictionary_client::foreach<dd::Schema>(dd::Object_key const*, std::function<bool (std::unique_ptr<dd::Schema, std::default_delete<dd::Schema> >&)> const&) const [clone .constprop.0] () #14 0x0000000001359294 in dd::upgrade::do_server_upgrade_checks(THD*) () #15 0x0000000001254fa4 in dd::bootstrap::restart(THD*) () #16 0x000000000137ab74 in dd::upgrade_57::restart_dictionary(THD*) () #17 0x000000000138b718 in dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*) () #18 0x0000000000bcafe4 in handle_bootstrap () #19 0x00000000016d4d40 in pfs_spawn_thread () #20 0x0000400033bec230 in start_thread () from target:/lib64/libpthread.so.0 #21 0x0000400033de77dc in thread_start () from target:/lib64/libc.so.6 How to repeat: 1. Create a MySQL server 8.0.40 2. Create 1 Million tables in a schema. 3. Upgrade to 8.0.41 this should be quick. 4. Try to upgrade to 8.0.42 - issue will be reproduced.