Bug #98050 Deadlock on LOCK_global_system_variables, LOCK_plugin, LOCK_system_variables_has
Submitted: 23 Dec 2019 8:13 Modified: 6 Oct 7:43
Reporter: Eduardo Ortega Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Clone Plugin Severity:S2 (Serious)
Version:8.0.18 OS:CentOS (7)
Assigned to: Marc Alff CPU Architecture:x86

[23 Dec 2019 8:13] Eduardo Ortega
Description:
# What we did

We have a script reconciling configuration from the /etc/my.cnf with actual running configuration. Part of what it does is load plugins when they are defined in the configuration but not loaded already. This script attempted to run this:

INSTALL PLUGIN mysql_clone SONAME 'mysql_clone.so'

And the server hung. We managed to attach gdb to the process and inspect the issue - see below

# What we wanted to happen

We wanted the plugin to be installed and the server to continue to work without issues

# What actually happened

The server hang. Now new connections could be established. We examined the problem with gdb, our findings are attached. From them, we believe there is a deadlock involving at least LOCK_global_system_variables, LOCK_plugin and LOCK_system_variables_hash.

How to repeat:
We have only seen this under load in MySQL 8.0.18 (though it might be that it is the only version we are currently running with the CLONE plugin). So far, we have not been able to reproduce it reliably.
[23 Dec 2019 8:14] Eduardo Ortega
Investigation from backtrace & attached gdb

Attachment: investigation_on_sys_deadlock.txt (text/plain), 11.45 KiB.

[23 Dec 2019 9:16] MySQL Verification Team
Verified!

mysql> show processlist;
+-------+-----------------+-----------+------+---------+------+------------------------+----------------------------------------------+
| Id    | User            | Host      | db   | Command | Time | State                  | Info                                         |
+-------+-----------------+-----------+------+---------+------+------------------------+----------------------------------------------+
|     4 | event_scheduler | localhost | NULL | Daemon  |  584 | Waiting on empty queue | NULL                                         |
|  2524 | root            | localhost | NULL | Query   |    0 | starting               | show processlist                             |
| 24439 | root            | localhost | NULL | Query   |  126 | executing              | show global variables                        |
| 24441 | root            | localhost | NULL | Connect |  126 | login                  | NULL                                         |
| 24442 | root            | localhost | NULL | Query   |  126 | System lock            | install plugin clone soname 'mysql_clone.so' |
+-------+-----------------+-----------+------+---------+------+------------------------+----------------------------------------------+
5 rows in set (0.00 sec)

mysql> select version();
+-----------+
| version() |
+-----------+
| 8.0.18    |
+-----------+
1 row in set (0.00 sec)
[23 Dec 2019 15:56] MySQL Verification Team
affects current trunk build today also.

Attachment: bug98050_trunk.txt (text/plain), 142.59 KiB.

[6 Oct 7:43] MySQL Verification Team
turns out this isn't fixed yet.  Reopening report.