Bug #98050 Deadlock on LOCK_global_system_variables, LOCK_plugin, LOCK_system_variables_has

• Submitted: 23 Dec 2019 8:13
• Modified: 6 Oct 2020 7:43
• Reporter: Eduardo Ortega
• Status: Verified
• Category: MySQL Server: Clone Plugin
• Severity: S2 (Serious)
• Version: 8.0.18
• OS: CentOS (7)
• Assigned to: Marc ALFF
• CPU Architecture: x86

[23 Dec 2019 8:13] Eduardo Ortega

Description:
# What we did

We have a script reconciling configuration from the /etc/my.cnf with actual running configuration. Part of what it does is load plugins when they are defined in the configuration but not loaded already. This script attempted to run this:

INSTALL PLUGIN mysql_clone SONAME 'mysql_clone.so'

And the server hung. We managed to attach gdb to the process and inspect the issue - see below

# What we wanted to happen

We wanted the plugin to be installed and the server to continue to work without issues

# What actually happened

The server hang. Now new connections could be established. We examined the problem with gdb, our findings are attached. From them, we believe there is a deadlock involving at least LOCK_global_system_variables, LOCK_plugin and LOCK_system_variables_hash.

How to repeat:
We have only seen this under load in MySQL 8.0.18 (though it might be that it is the only version we are currently running with the CLONE plugin). So far, we have not been able to reproduce it reliably.

[23 Dec 2019 8:14] Eduardo Ortega

Investigation from backtrace & attached gdb

Attachment: investigation_on_sys_deadlock.txt (text/plain), 11.45 KiB.

[23 Dec 2019 9:16] MySQL Verification Team

Verified!

mysql> show processlist;
+-------+-----------------+-----------+------+---------+------+------------------------+----------------------------------------------+
| Id    | User            | Host      | db   | Command | Time | State                  | Info                                         |
+-------+-----------------+-----------+------+---------+------+------------------------+----------------------------------------------+
|     4 | event_scheduler | localhost | NULL | Daemon  |  584 | Waiting on empty queue | NULL                                         |
|  2524 | root            | localhost | NULL | Query   |    0 | starting               | show processlist                             |
| 24439 | root            | localhost | NULL | Query   |  126 | executing              | show global variables                        |
| 24441 | root            | localhost | NULL | Connect |  126 | login                  | NULL                                         |
| 24442 | root            | localhost | NULL | Query   |  126 | System lock            | install plugin clone soname 'mysql_clone.so' |
+-------+-----------------+-----------+------+---------+------+------------------------+----------------------------------------------+
5 rows in set (0.00 sec)

mysql> select version();
+-----------+
| version() |
+-----------+
| 8.0.18    |
+-----------+
1 row in set (0.00 sec)

[23 Dec 2019 15:56] MySQL Verification Team

affects current trunk build today also.

Attachment: bug98050_trunk.txt (text/plain), 142.59 KiB.

[6 Oct 2020 7:43] MySQL Verification Team

turns out this isn't fixed yet.  Reopening report.