Bug #86230 slave_compressed_protocol doesn't work with semi-sync replication in MySQL-5.7
Submitted: 8 May 2017 23:35 Modified: 5 Oct 2017 13:36
Reporter: Xiaofeng Dong Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Replication Severity:S2 (Serious)
Version:5.7.18 OS:Any
Assigned to: CPU Architecture:Any
Tags: compression, protocol, replication, semisynchronous

[8 May 2017 23:35] Xiaofeng Dong
Description:
In MySQL-5.6, the slave_compressed_protocol option can work with semisynchronous replication.

As of MySQL-5.7.4, semisynchronous replication added a separate thread to receive ACK.

But the ack_receiver thread does not consider the compression protocol, it can't unpack a compressed ACK correctly.

How to repeat:
########################################
source include/have_semisync_plugin.inc;
source include/not_embedded.inc;
source include/have_innodb.inc;
source include/master-slave.inc;

let $engine_type= InnoDB;

--echo # 1. async repl

connection master;
eval create table t1 (a int) engine=$engine_type;
--source include/sync_slave_sql_with_master.inc
connection slave;
source include/stop_slave.inc;

--echo # 2. enable semi-sync plugins on master

connection master;
eval INSTALL PLUGIN rpl_semi_sync_master SONAME '$SEMISYNC_MASTER_PLUGIN';
set global rpl_semi_sync_master_timeout= 5000;
set global rpl_semi_sync_master_enabled = 1;
show variables like 'rpl_semi_sync_master_enabled';

--echo # 3. install semi-sync plugins on slave

connection slave;
eval INSTALL PLUGIN rpl_semi_sync_slave SONAME '$SEMISYNC_SLAVE_PLUGIN';
set global rpl_semi_sync_slave_enabled = 1;
set global slave_compressed_protocol = 1;
source include/start_slave.inc;

--echo # 4. test compression with semi-sync plugins

connection master;
disable_query_log;
let $i= 5;
while ($i)
{
  eval insert into t1 values ($i);
  dec $i;
}
enable_query_log;
--source include/sync_slave_sql_with_master.inc

connection slave;
select count(*) from t1;

--echo # 5. cleanup

connection master;
drop table t1;
--source include/sync_slave_sql_with_master.inc

connection slave;
source include/stop_slave.inc;
set global rpl_semi_sync_slave_enabled = 0;
disable_warnings;
error 0,1305;
UNINSTALL PLUGIN rpl_semi_sync_slave;
enable_warnings;
connection master;
set global rpl_semi_sync_master_enabled = 0;
--source include/stop_dump_threads.inc
disable_warnings;
error 0,1305;
UNINSTALL PLUGIN rpl_semi_sync_master;
enable_warnings;

connection slave;
set global slave_compressed_protocol = 0;
source include/start_slave.inc;
connection master;
--source include/rpl_end.inc
########################################

Testing this case in mysql-5.6.36 will be passed.

but in mysql-5.7.18, we will get:

mysql-test-run: WARNING: Got errors/warnings while running tests, please examine 'var/log/warnings' for details.

in var/log/warnings:

...

[ERROR] Read semi-sync reply magic number error

[ERROR] Got timeout reading communication packets

[ERROR] Got an error reading communication packets

...

Suggested fix:
plugin/semisync/semisync_master_ack_receiver.cc

void Ack_receiver::run()
{

...

    while (i < m_slaves.size())
    {
      if (listener.is_socket_active(i))
      {
        ulong len;

        net_clear(&net, 0);
        net.vio= &m_slaves[i].vio;

        /* set net.compress */
        net.compress = *(&m_slaves[i].thd->net.compress);    /* but now net is a private member of thd */

        len= my_net_read(&net);
        if (likely(len != packet_error))
          repl_semisync.reportReplyPacket(m_slaves[i].server_id(),
                                          net.read_pos, len);
        else if (net.last_errno == ER_NET_READ_ERROR)
          listener.clear_socket_info(i);
      }
      i++;
    }

...

}
[8 May 2017 23:37] Xiaofeng Dong
test case

Attachment: rpl_semi_sync_compression.test (application/octet-stream, text), 1.75 KiB.

[9 May 2017 5:42] Umesh Shastry
Hello Weaver Dong,

Thank you for the report and test case.

Thanks,
Umesh
[9 May 2017 5:43] Umesh Shastry
## I'm not sure whether this is intended behavior, but observed that 5.7.18 generating warnings whereas none in 5.6.36

[umshastr@hod03]/export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test: ./mtr rpl_semi_sync_compression
Logging: ./mtr  rpl_semi_sync_compression
MySQL Version 5.7.18
Checking supported features...
 - SSL connections supported
Collecting tests...
Checking leftover processes...
Removing old var directory...
Creating var directory '/export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test/var'...
Installing system database...
Using parallel: 1

==============================================================================

TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009
include/master-slave.inc
Warnings:
Note    ####    Sending passwords in plain text without SSL/TLS is extremely insecure.
Note    ####    Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
[connection master]
# 1. async repl
create table t1 (a int) engine=InnoDB;
include/sync_slave_sql_with_master.inc
include/stop_slave.inc
# 2. enable semi-sync plugins on master
INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';
set global rpl_semi_sync_master_timeout= 5000;
set global rpl_semi_sync_master_enabled = 1;
show variables like 'rpl_semi_sync_master_enabled';
Variable_name   Value
rpl_semi_sync_master_enabled    ON
# 3. install semi-sync plugins on slave
INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';
set global rpl_semi_sync_slave_enabled = 1;
set global slave_compressed_protocol = 1;
include/start_slave.inc
# 4. test compression with semi-sync plugins
include/sync_slave_sql_with_master.inc
select count(*) from t1;
count(*)
5
# 5. cleanup
drop table t1;
include/sync_slave_sql_with_master.inc
include/stop_slave.inc
set global rpl_semi_sync_slave_enabled = 0;
UNINSTALL PLUGIN rpl_semi_sync_slave;
set global rpl_semi_sync_master_enabled = 0;
include/stop_dump_threads.inc
UNINSTALL PLUGIN rpl_semi_sync_master;
set global slave_compressed_protocol = 0;
include/start_slave.inc
include/rpl_end.inc
main.rpl_semi_sync_compression           [ pass ]  11306
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 11.306 of 25 seconds executing testcases
mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test/var/log/warnings' for details.

Completed: All 1 tests were successful.

[umshastr@hod03]/export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test: cat var/log/warnings
main.rpl_semi_sync_compression
line
2017-05-09T05:38:15.269582Z 0 [ERROR] Read semi-sync reply magic number error
2017-05-09T05:38:15.270663Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got timeout reading communication packets
2017-05-09T05:38:20.269314Z 8 [Warning] Timeout waiting for reply of binlog (file: master-bin.000001, pos: 583), semi-sync up to file master-bin.000001, position 330.
2017-05-09T05:38:20.270274Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got an error reading communication packets
2017-05-09T05:38:20.271624Z 10 [ERROR] Semi-sync master failed on net_flush() before waiting for slave reply
2017-05-09T05:38:23.276785Z 0 [ERROR] Read semi-sync reply magic number error
2017-05-09T05:38:23.277886Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got timeout reading communication packets
2017-05-09T05:38:23.284905Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got an error reading communication packets
2017-05-09T05:38:25.289100Z 0 [ERROR] Read semi-sync reply magic number error
2017-05-09T05:38:25.290180Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got timeout reading communication packets
2017-05-09T05:38:25.294998Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got an error reading communication packets
^ Found warnings in /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test/var/log/mysqld.1.err
ok
[9 May 2017 5:45] Umesh Shastry
-- 5.6.36

[umshastr@hod03]/export/umesh/server/binaries/GABuilds/mysql-5.6.36/mysql-test: ./mtr rpl_semi_sync_compression
Logging: ./mtr  rpl_semi_sync_compression
2017-05-09 07:40:12 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
2017-05-09 07:40:12 0 [Note] /export/umesh/server/binaries/GABuilds/mysql-5.6.36/bin/mysqld (mysqld 5.6.36) starting as process 14377 ...
2017-05-09 07:40:12 14377 [Note] Plugin 'FEDERATED' is disabled.
2017-05-09 07:40:12 14377 [Note] Binlog end
2017-05-09 07:40:12 14377 [Note] Shutting down plugin 'CSV'
2017-05-09 07:40:12 14377 [Note] Shutting down plugin 'MyISAM'
MySQL Version 5.6.36
Checking supported features...
 - SSL connections supported
Collecting tests...
Checking leftover processes...
Removing old var directory...
Creating var directory '/export/umesh/server/binaries/GABuilds/mysql-5.6.36/mysql-test/var'...
Installing system database...
Using parallel: 1

==============================================================================

TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009
include/master-slave.inc
Warnings:
Note    ####    Sending passwords in plain text without SSL/TLS is extremely insecure.
Note    ####    Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
[connection master]
# 1. async repl
create table t1 (a int) engine=InnoDB;
include/sync_slave_sql_with_master.inc
include/stop_slave.inc
# 2. enable semi-sync plugins on master
INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';
set global rpl_semi_sync_master_timeout= 5000;
set global rpl_semi_sync_master_enabled = 1;
show variables like 'rpl_semi_sync_master_enabled';
Variable_name   Value
rpl_semi_sync_master_enabled    ON
# 3. install semi-sync plugins on slave
INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';
set global rpl_semi_sync_slave_enabled = 1;
set global slave_compressed_protocol = 1;
include/start_slave.inc
# 4. test compression with semi-sync plugins
include/sync_slave_sql_with_master.inc
select count(*) from t1;
count(*)
5
# 5. cleanup
drop table t1;
include/sync_slave_sql_with_master.inc
include/stop_slave.inc
set global rpl_semi_sync_slave_enabled = 0;
UNINSTALL PLUGIN rpl_semi_sync_slave;
set global rpl_semi_sync_master_enabled = 0;
include/stop_dump_threads.inc
UNINSTALL PLUGIN rpl_semi_sync_master;
set global slave_compressed_protocol = 0;
include/start_slave.inc
include/rpl_end.inc
main.rpl_semi_sync_compression           [ pass ]    212
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 0.212 of 7 seconds executing testcases

Completed: All 1 tests were successful.

[umshastr@hod03]/export/umesh/server/binaries/GABuilds/mysql-5.6.36/mysql-test:
[5 Oct 2017 13:36] Margaret Fisher
Posted by developer:
 
Changelog entry added for MySQL 8.0.4 and 5.7.21:

The receiver thread for semisynchronous replication was not able to receive acknowledgements from slaves that used compression of the master/slave protocol  (slave_compressed_protocol = ON).  The receiver thread now handles compressed acknowledgements correctly.