Bug #80394 incorrect semi-sync replication status handling
Submitted: 16 Feb 2016 12:37 Modified: 7 Mar 2016 9:33
Reporter: Matthew Lord Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server: Replication Severity:S2 (Serious)
Version:5.7.11 OS:Any
Assigned to: CPU Architecture:Any

[16 Feb 2016 12:37] Matthew Lord
Description:
As you'll see when repeating the steps below, the global rpl_semi_sync_master_enabled variable is not correctly reflected as 0/OFF after semi-sync replication has been switched OFF due to a master timeout. 

You can see the behavior clearly reflected in the INSERT statement behavior, as well as in the MySQL error log output.  

As shown below, in order to re-enable semi-sync replication after a master timeout, you must manually turn it OFF and then back ON again. 

This was verified on OL 7.2 x86_64, using MySQL 5.7.11-community. 

How to repeat:
mkdir /tmp/bug
chown mysql:mysql /tmp/bug

mysqld --no-defaults --initialize-insecure --datadir=/tmp/bug --user=mysql  

mysqld --no-defaults --user=mysql --port=3399 --socket=/tmp/bug/mysql.sock --datadir=/tmp/bug --log-bin --server-id=1 &

mysql --no-defaults --socket=/tmp/bug/mysql.sock -e

create database bug;
use bug;
create table ssbug (id int not null primary key, name varchar(100));

INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';
SET GLOBAL rpl_semi_sync_master_enabled=ON;
show global variables like "rpl%";

insert into ssbug (id, name) values (1, "matt");

show global variables like "rpl%";

insert into ssbug (id, name) values (2, "lily");
insert into ssbug (id, name) values (3, "sid");

SET GLOBAL rpl_semi_sync_master_enabled=OFF;
SET GLOBAL rpl_semi_sync_master_enabled=ON;

insert into ssbug (id, name) values (4, "sid");

Suggested fix:
We need to properly set and reflect the global variable rpl_semi_sync_master_enabled=OFF when we've disabled semi-sync replication due to hitting a master timeout. 

The user should then be able to re-enable it simply by turning it on again:
SET GLOBAL rpl_semi_sync_master_enabled=ON;
[16 Feb 2016 12:44] Matthew Lord
Sorry, just noticed an error in the "how to repeat" steps. The mysql client call has an errant trailing "-e". That step should simply be:

mysql --no-defaults --socket=/tmp/bug/mysql.sock
[7 Mar 2016 9:33] Sujatha Sivakumar
Analysis:
---------
'rpl_semi_sync_master_enabled' global variable controls whether 
semisynchronous plugin is enabled or disabled on the master. If it is ON
plugin is enabled on the master and if it is OFF plugin is disabled on master.

There is another important variable which represents master status.

i.e Rpl_semi_sync_master_status.

This variable specifies whether semisynchronous replication currently is operational on the master or not. The value is ON if the plugin has been enabled and a commit acknowledgment has occurred. It is OFF if the plugin is not enabled or the master has fallen back to asynchronous replication due to commit acknowledgment timeout.

In the event of timeout following things will happen:
        
master> SHOW GLOBAL STATUS LIKE 'rpl%status';
+-----------------------------+-------+
| Variable_name               | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_status | OFF   |
+-----------------------------+-------+

MySQL error log will report the situation,but not as an error:

120617 18:53:09 [Warning] Timeout waiting for reply of binlog (file:
alpha-bin.000004, pos: 357), semi-sync up to file , position 0.
120617 18:53:09 [Note] Semi-sync replication switched OFF.

If a timeout occurs the master automatically reverts to asynchronous
replication for future transactions. But this change in reflected by 'rpl_semi_sync_master_status' as shown above. The semi sync plugin is still enabled on the master.

Hence rpl_semi_sync_master_enabled will be 'ON'.

Please note that the master will automatically switch back to semisynchronous replication, when at least one semisynchronous slave catches up, the master returns to semisynchronous replication. Semisync plugin does this internally without user's intervention.

Hence the existing behavior of rpl_semi_sync_master_enabled is fine and
we don't have to fix anything as part of this bug.