Bug #121302 Server hangs: INSTALL PLUGIN (I_S plugin) re-locks LOCK_plugin via DD attachable transaction
Submitted: 17 Sep 6:34 Modified: 20 Sep 2:31
Reporter: Shi W Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Server: Information schema Severity:S1 (Critical)
Version:8.4.11;8.0.46;8.0.44;8.0.36 OS:Any
Assigned to: CPU Architecture:x86
Tags: hang, information schema, plugin

[17 Sep 6:34] Shi W
Description:
INSTALL PLUGIN for a MYSQL_INFORMATION_SCHEMA_PLUGIN can permanently self-deadlock the executing thread on LOCK_plugin. Since the lock is then held forever, every new connection and every disconnect blocks, and shutdown hangs too.
  
mysql_install_plugin() holds LOCK_plugin while calling store_dynamic_plugin_I_S_metadata(), which acquires the information_schema dd::Schema object. On a DD cache miss that read happens inside an attachable 
read-only transaction, whose destructor unconditionally calls ha_close_connection() → plugin_foreach() → mysql_mutex_lock(&LOCK_plugin) on the same thread. LOCK_plugin is not recursive.

We reproduced this bug on 8.0 and 8.4 servers.

Deadlock cycle (function names; line numbers from 8.0.46 source):
```
mysql_install_plugin            sql_plugin.cc:2439   <-- holds LOCK_plugin
  store_dynamic_plugin_I_S_metadata     dd/info_schema/metadata.cc:648
    Update_context::Update_context             dd/info_schema/metadata.cc:114
      Dictionary_client::acquire<dd::Schema>("information_schema")
        Shared_dictionary_cache::get_uncached     <-- only on a cache MISS
          Storage_adapter::get<Item_name_key,dd::Schema>  storage_adapter.cc:211
            THD::Attachable_trx::~Attachable_trx  sql_class.cc:516
              ha_close_connection                 handler.cc:954
                plugin_foreach_with_mask          sql_plugin.cc:2709
                  mysql_mutex_lock(&LOCK_plugin)  <-- DEADLOCK
```

How to repeat:
Two preconditions: install the I_S sub-plugin, and evict information_schema from the DD cache; schema_definition_cache = 256

```bash
# 1) create 300 schemas, each with a table
for i in $(seq 1 300); do
  echo "CREATE DATABASE IF NOT EXISTS db_$i; CREATE TABLE IF NOT EXISTS db_$i.t(a INT);"
done | mysql -uroot
  
# 2) acquire each schema once (opening a table acquires its schema),
#    which LRU-evicts 'information_schema'
for i in $(seq 1 300); do echo "SELECT 1 FROM db_$i.t LIMIT 0;"; done | mysql -uroot
  
# 3) trigger
mysql -uroot -e "INSTALL PLUGIN CONNECTION_CONTROL_FAILED_LOGIN_ATTEMPTS \
                   SONAME 'connection_control.so';"
```
Actual result:
Release build: INSTALL PLUGIN never returns; new connections hang; disconnects hang; mysqladmin shutdown hangs; SIGKILL required. 

Debug build aborts immediately with:  
  safe_mutex: Trying to lock mutex at sql/sql_plugin.cc, line 2709,
              when the mutex was already locked at sql/sql_plugin.cc, line 1343 in thread T@11

Suggested fix:
Don't hold LOCK_plugin across the DD metadata update. The same function already unlocks LOCK_plugin + LOCK_system_variables_hash a few lines earlier (line 2379) to take the MDL lock and re-locks at 2422 — so the DD call can be moved outside the critical section the same way. The UNINSTALL counterpart (remove_dynamic_plugin_I_S_metadata()) should be audited for the same pattern.
[17 Sep 7:51] Shane Bester
related: https://bugs.mysql.com/bug.php?id=98050
[20 Sep 2:31] Shi W
will it be fixed?