Bug #121025 check_table_funs retains opened tables; 1M-table upgrade OOMs with #117983 fix
Submitted: 28 Jul 21:38
Reporter: Sanyam Jain Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Server: Data Dictionary Severity:S3 (Non-critical)
Version:8.4.9 OS:Any
Assigned to: CPU Architecture:Any
Tags: data dictionary, OOM, upgrade

[28 Jul 21:38] Sanyam Jain
Description:
Bug #119037 reported an OOM during upgrade with about 1 million tables and was closed as a duplicate of bug #117983. I retested the #119037 scenario with the announced #117983 fix applied. The fix reduces memory use, but it does not resolve this OOM. The two reports involve different memory owners.

Bug #117983 covers memory allocated from the upgrade THD's MEM_ROOT while validating views, routines, events, and other data-dictionary objects. Its fix clears that arena between objects and adds a temporary query arena around the upgrade checks.

This report covers TABLE and handler objects opened by `check_table_funs()`. The function calls `open_table()` for every table whose definition uses a CHECK constraint, generated column, default expression, or partition expression. It does not close those tables. They remain on `thd->open_tables` until the upgrade phase ends. These objects are not released by clearing MEM_ROOT, and the #117983 fix does not add cleanup to `check_table_funs()`.

Source path:
https://github.com/mysql/mysql-server/blob/mysql-8.4.9/sql/dd/impl/upgrade/server.cc#L796-...

I created 1M tables on a 32 GiB host v8.0.40, built a target by applying the fix(provided in #117983) to 8.4.9 source and ran the upgrade.

mysqld was OOM-killed by the kernel about 34 minutes into the upgrade, mid-check, at 29.5 GB RSS and climbing linearly:

[397603.419675] systemd-userwor invoked oom-killer: gfp_mask=0x140cca(...), order=0
[397603.512653] Out of memory: Killed process 200937 (mysqld) total-vm:32582976kB,
                anon-rss:29460468kB, file-rss:0kB, shmem-rss:0kB, pgtables:58096kB

How to repeat:
(Same steps as reported in #119037)

1. Create a MySQL server 8.0.40
2. Create 1 Million tables in a schema.
3. Upgrade to Target server - MySQL Community Server 8.4.9 built from source, with the announced #117983 fix applied.
4. Run the upgrade - issue will be reproduced.

Suggested fix:
Release each table opened by `check_table_funs()` after that probe finishes. The
cleanup must be scoped to the resources created by the probe.

Record the current `thd->open_tables` pointer before calling `open_table()`. After the
probe, close entries only until the list returns to that saved watermark.