MySQL Bugs: #88042: upgrade to 8.0 hits stack overflow

Bug #88042	upgrade to 8.0 hits stack overflow
Submitted:	10 Oct 2017 8:22	Modified:	16 Oct 2017 17:47
Reporter:	Shane Bester (Platinum Quality Contributor)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Data Dictionary	Severity:	S2 (Serious)
Version:	8.0.4	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
On first start of 8.0 during upgrade, I hit a stack overflow crash:

[Note] [000000] InnoDB: 8.0.4 started; log sequence number 0
[Note] [000000] InnoDB: Loading buffer pool(s) from ib_buffer_pool
[Note] [003960] Created Data Dictionary for upgrade
[Note] [000000] InnoDB: Buffer pool(s) load completed at 171010 10:15:38
[Note] [000000] Finished populating Data Dictionary tables with data.
[Note] [000000] Finished migrating TABLE statistics data.
[Note] [000000] Finished migrating INDEX statistics data.
08:18:12 UTC - mysqld got exception 0xc00000fd ;

ucrtbased.dll!__crt_fast_decode_pointer
ucrtbased.dll!__acrt_FlsGetValue
ucrtbased.dll!__crt_state_management::dual_state_global<int>::value
ucrtbased.dll!_query_new_mode
ucrtbased.dll!_calloc_base
ucrtbased.dll!_malloc_dbg
ucrtbased.dll!malloc
mysqld-debug.exe!ut_allocator<unsigned char>::allocate
mysqld-debug.exe!mem_heap_create_block_func
mysqld-debug.exe!mem_heap_create_func
mysqld-debug.exe!dict_load_table_one

<cut ~500 frames>

mysqld-debug.exe!dict_load_table_one
mysqld-debug.exe!dict_load_table
mysqld-debug.exe!dict_table_get_low
mysqld-debug.exe!dd_upgrade_drop_sys_tables
mysqld-debug.exe!dd_upgrade_finish
mysqld-debug.exe!dd::upgrade::ha_finish_upgrade
mysqld-debug.exe!plugin_foreach_with_mask
mysqld-debug.exe!plugin_foreach_with_mask
mysqld-debug.exe!dd::upgrade::ha_finish_upgrade
mysqld-debug.exe!dd::upgrade::finalize_upgrade

How to repeat:
On a clean 5.7,  import the attached test.sql to create a test schema.
Shutdown 5.7.
Start 8.0 on the datadir.

create the schema

Attachment: test.sql (application/octet-stream, text), 473.91 KiB.

Posted by developer:
 
During upgrade process, InnoDB system table is moved to LRU list.
When there are huge number of tables, InnoDB is evicting tables as part of table cache cleanup (done by master thread).

Once a system table is evicted, it cannot be loaded and goes into recursive mode trying to load system table.

This patch should fix it:

diff --git a/storage/innobase/dict/dict0dict.cc b/storage/innobase/dict/dict0dict.cc
index c4dc5d1c..e28eb21 100644
--- a/storage/innobase/dict/dict0dict.cc
+++ b/storage/innobase/dict/dict0dict.cc
@@ -7832,10 +7832,6 @@ dict_table_change_id_sys_tables()
 	for (uint32_t i = 0; i < SYS_NUM_SYSTEM_TABLES; i++) {
 		dict_table_t*	system_table = dict_table_get_low(SYSTEM_TABLE_NAME[i]);
 
-		/* It's possible the SYS_VIRTUAL is not exist. */
-		if (system_table == nullptr && i == 8) {
-			continue;
-		}
 		ut_a(system_table != nullptr);
 		ut_ad(dict_sys_table_id[i] == system_table->id);
 
@@ -7847,6 +7843,8 @@ dict_table_change_id_sys_tables()
 		dict_table_change_id_in_cache(system_table, new_table_id);
 
 		dict_sys_table_id[i] = system_table->id;
+
+		dict_table_prevent_eviction(system_table);
 	}
 }

Posted by developer:
 
regression of Bug#26757171 - INNODB DICTIONARY TABLES SHOULD BE REMOVED AT THE END OF UPGRADE

But this is pushed only 8.0.4.. I wonder how this is seen in 8.0.3

Posted by developer:
 
Fixed as of the upcoming 8.0.4, 9.0.0 release, and here's the changelog entry:

 A stack overflow error was encountered on startup after upgrading to
MySQL 8.0.4 due to repeated attempts to load an evicted InnoDB system
table.