Bug #71182 uninstall semisync plugin may lead to hang
Submitted: 19 Dec 2013 11:35 Modified: 20 Dec 2013 19:05
Reporter: zhai weixiang (OCA) Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Replication Severity:S3 (Non-critical)
Version:5.6.15 OS:Any
Assigned to: CPU Architecture:Any

[19 Dec 2013 11:35] zhai weixiang
Description:

1.configure semisync (make sure rpl_semi_sync_master_timeout is set to a very large value)

on master:
root@(none) 07:15:49>show variables like '%semi%';
+------------------------------------+-----------+
| Variable_name                      | Value     |
+------------------------------------+-----------+
| rpl_semi_sync_master_enabled       | ON        |
| rpl_semi_sync_master_timeout       | 100000000 |
| rpl_semi_sync_master_trace_level   | 32        |
| rpl_semi_sync_master_wait_no_slave | ON        |
| rpl_semi_sync_slave_enabled        | OFF       |
| rpl_semi_sync_slave_trace_level    | 32        |
+------------------------------------+-----------+
6 rows in set (0.00 sec)

on slave:
root@(none) 07:16:52>show variables like '%semi%';
+---------------------------------+-------+
| Variable_name                   | Value |
+---------------------------------+-------+
| rpl_semi_sync_slave_enabled     | ON    |
| rpl_semi_sync_slave_trace_level | 32    |
+---------------------------------+-------+
2 rows in set (0.00 sec)

2. run workload on master, and then stop the io_thread on slave;

Because rpl_semi_sync_master_wait_no_slave is set to ON, then all user threads stall at:
ReplSemiSyncMaster::cond_timewait,ReplSemiSyncMaster::commitTrx

3.uninstall semisync plugin on master;

root@(none) 07:08:56>uninstall plugin rpl_semi_sync_master;
Query OK, 0 rows affected, 1 warning (0.00 sec)

root@(none) 07:09:01>
root@(none) 07:09:02>show warnings;
+---------+------+----------------------------------------------------+
| Level   | Code | Message                                            |
+---------+------+----------------------------------------------------+
| Warning | 1620 | Plugin is busy and will be uninstalled on shutdown |
+---------+------+----------------------------------------------------+
1 row in set (0.00 sec)

4.start io_thread on slave. Since the semisync_master_plugin was uninstalled (but not completely), it will be failed to connect to master, And the error message:

2013-12-19 19:22:29 21760 [ERROR] Error reading packet from server: Failed to run hook 'reserve_header' ( server_errno=1105)

5. I can't wake up the user threads by setting rpl_semi_sync_master_enabled

root@(none) 07:23:39>set global rpl_semi_sync_master_enabled  = 0;
ERROR 1193 (HY000): Unknown system variable 'rpl_semi_sync_master_enabled'

Since the value of rpl_semi_sync_master_timeout is very large,  I have to restart the server to kill these threads.

How to repeat:
described above.

Suggested fix:
wake up all user threads while uninstall the plugin.
[20 Dec 2013 19:05] Sveta Smirnova
Thank you for the report.

Strictly speaking this is wrong usage scenario, but I agree there is no way to solve it without server restart or very long wait. Verified as described.