| Bug #86230 | slave_compressed_protocol doesn't work with semi-sync replication in MySQL-5.7 | ||
|---|---|---|---|
| Submitted: | 8 May 2017 23:35 | Modified: | 5 Oct 2017 13:36 |
| Reporter: | Xiaofeng Dong | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Replication | Severity: | S2 (Serious) |
| Version: | 5.7.18 | OS: | Any |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | compression, protocol, replication, semisynchronous | ||
[8 May 2017 23:37]
Xiaofeng Dong
test case
Attachment: rpl_semi_sync_compression.test (application/octet-stream, text), 1.75 KiB.
[9 May 2017 5:42]
MySQL Verification Team
Hello Weaver Dong, Thank you for the report and test case. Thanks, Umesh
[9 May 2017 5:43]
MySQL Verification Team
## I'm not sure whether this is intended behavior, but observed that 5.7.18 generating warnings whereas none in 5.6.36 [umshastr@hod03]/export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test: ./mtr rpl_semi_sync_compression Logging: ./mtr rpl_semi_sync_compression MySQL Version 5.7.18 Checking supported features... - SSL connections supported Collecting tests... Checking leftover processes... Removing old var directory... Creating var directory '/export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test/var'... Installing system database... Using parallel: 1 ============================================================================== TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009 include/master-slave.inc Warnings: Note #### Sending passwords in plain text without SSL/TLS is extremely insecure. Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. [connection master] # 1. async repl create table t1 (a int) engine=InnoDB; include/sync_slave_sql_with_master.inc include/stop_slave.inc # 2. enable semi-sync plugins on master INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so'; set global rpl_semi_sync_master_timeout= 5000; set global rpl_semi_sync_master_enabled = 1; show variables like 'rpl_semi_sync_master_enabled'; Variable_name Value rpl_semi_sync_master_enabled ON # 3. install semi-sync plugins on slave INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so'; set global rpl_semi_sync_slave_enabled = 1; set global slave_compressed_protocol = 1; include/start_slave.inc # 4. test compression with semi-sync plugins include/sync_slave_sql_with_master.inc select count(*) from t1; count(*) 5 # 5. cleanup drop table t1; include/sync_slave_sql_with_master.inc include/stop_slave.inc set global rpl_semi_sync_slave_enabled = 0; UNINSTALL PLUGIN rpl_semi_sync_slave; set global rpl_semi_sync_master_enabled = 0; include/stop_dump_threads.inc UNINSTALL PLUGIN rpl_semi_sync_master; set global slave_compressed_protocol = 0; include/start_slave.inc include/rpl_end.inc main.rpl_semi_sync_compression [ pass ] 11306 -------------------------------------------------------------------------- The servers were restarted 0 times Spent 11.306 of 25 seconds executing testcases mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test/var/log/warnings' for details. Completed: All 1 tests were successful. [umshastr@hod03]/export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test: cat var/log/warnings main.rpl_semi_sync_compression line 2017-05-09T05:38:15.269582Z 0 [ERROR] Read semi-sync reply magic number error 2017-05-09T05:38:15.270663Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got timeout reading communication packets 2017-05-09T05:38:20.269314Z 8 [Warning] Timeout waiting for reply of binlog (file: master-bin.000001, pos: 583), semi-sync up to file master-bin.000001, position 330. 2017-05-09T05:38:20.270274Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got an error reading communication packets 2017-05-09T05:38:20.271624Z 10 [ERROR] Semi-sync master failed on net_flush() before waiting for slave reply 2017-05-09T05:38:23.276785Z 0 [ERROR] Read semi-sync reply magic number error 2017-05-09T05:38:23.277886Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got timeout reading communication packets 2017-05-09T05:38:23.284905Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got an error reading communication packets 2017-05-09T05:38:25.289100Z 0 [ERROR] Read semi-sync reply magic number error 2017-05-09T05:38:25.290180Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got timeout reading communication packets 2017-05-09T05:38:25.294998Z 0 [ERROR] /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/bin/mysqld: Got an error reading communication packets ^ Found warnings in /export/umesh/server/binaries/GABuilds/mysql-advanced-5.7.18/mysql-test/var/log/mysqld.1.err ok
[9 May 2017 5:45]
MySQL Verification Team
-- 5.6.36 [umshastr@hod03]/export/umesh/server/binaries/GABuilds/mysql-5.6.36/mysql-test: ./mtr rpl_semi_sync_compression Logging: ./mtr rpl_semi_sync_compression 2017-05-09 07:40:12 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path. 2017-05-09 07:40:12 0 [Note] /export/umesh/server/binaries/GABuilds/mysql-5.6.36/bin/mysqld (mysqld 5.6.36) starting as process 14377 ... 2017-05-09 07:40:12 14377 [Note] Plugin 'FEDERATED' is disabled. 2017-05-09 07:40:12 14377 [Note] Binlog end 2017-05-09 07:40:12 14377 [Note] Shutting down plugin 'CSV' 2017-05-09 07:40:12 14377 [Note] Shutting down plugin 'MyISAM' MySQL Version 5.6.36 Checking supported features... - SSL connections supported Collecting tests... Checking leftover processes... Removing old var directory... Creating var directory '/export/umesh/server/binaries/GABuilds/mysql-5.6.36/mysql-test/var'... Installing system database... Using parallel: 1 ============================================================================== TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009 include/master-slave.inc Warnings: Note #### Sending passwords in plain text without SSL/TLS is extremely insecure. Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. [connection master] # 1. async repl create table t1 (a int) engine=InnoDB; include/sync_slave_sql_with_master.inc include/stop_slave.inc # 2. enable semi-sync plugins on master INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so'; set global rpl_semi_sync_master_timeout= 5000; set global rpl_semi_sync_master_enabled = 1; show variables like 'rpl_semi_sync_master_enabled'; Variable_name Value rpl_semi_sync_master_enabled ON # 3. install semi-sync plugins on slave INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so'; set global rpl_semi_sync_slave_enabled = 1; set global slave_compressed_protocol = 1; include/start_slave.inc # 4. test compression with semi-sync plugins include/sync_slave_sql_with_master.inc select count(*) from t1; count(*) 5 # 5. cleanup drop table t1; include/sync_slave_sql_with_master.inc include/stop_slave.inc set global rpl_semi_sync_slave_enabled = 0; UNINSTALL PLUGIN rpl_semi_sync_slave; set global rpl_semi_sync_master_enabled = 0; include/stop_dump_threads.inc UNINSTALL PLUGIN rpl_semi_sync_master; set global slave_compressed_protocol = 0; include/start_slave.inc include/rpl_end.inc main.rpl_semi_sync_compression [ pass ] 212 -------------------------------------------------------------------------- The servers were restarted 0 times Spent 0.212 of 7 seconds executing testcases Completed: All 1 tests were successful. [umshastr@hod03]/export/umesh/server/binaries/GABuilds/mysql-5.6.36/mysql-test:
[5 Oct 2017 13:36]
Margaret Fisher
Posted by developer: Changelog entry added for MySQL 8.0.4 and 5.7.21: The receiver thread for semisynchronous replication was not able to receive acknowledgements from slaves that used compression of the master/slave protocol (slave_compressed_protocol = ON). The receiver thread now handles compressed acknowledgements correctly.
Description: In MySQL-5.6, the slave_compressed_protocol option can work with semisynchronous replication. As of MySQL-5.7.4, semisynchronous replication added a separate thread to receive ACK. But the ack_receiver thread does not consider the compression protocol, it can't unpack a compressed ACK correctly. How to repeat: ######################################## source include/have_semisync_plugin.inc; source include/not_embedded.inc; source include/have_innodb.inc; source include/master-slave.inc; let $engine_type= InnoDB; --echo # 1. async repl connection master; eval create table t1 (a int) engine=$engine_type; --source include/sync_slave_sql_with_master.inc connection slave; source include/stop_slave.inc; --echo # 2. enable semi-sync plugins on master connection master; eval INSTALL PLUGIN rpl_semi_sync_master SONAME '$SEMISYNC_MASTER_PLUGIN'; set global rpl_semi_sync_master_timeout= 5000; set global rpl_semi_sync_master_enabled = 1; show variables like 'rpl_semi_sync_master_enabled'; --echo # 3. install semi-sync plugins on slave connection slave; eval INSTALL PLUGIN rpl_semi_sync_slave SONAME '$SEMISYNC_SLAVE_PLUGIN'; set global rpl_semi_sync_slave_enabled = 1; set global slave_compressed_protocol = 1; source include/start_slave.inc; --echo # 4. test compression with semi-sync plugins connection master; disable_query_log; let $i= 5; while ($i) { eval insert into t1 values ($i); dec $i; } enable_query_log; --source include/sync_slave_sql_with_master.inc connection slave; select count(*) from t1; --echo # 5. cleanup connection master; drop table t1; --source include/sync_slave_sql_with_master.inc connection slave; source include/stop_slave.inc; set global rpl_semi_sync_slave_enabled = 0; disable_warnings; error 0,1305; UNINSTALL PLUGIN rpl_semi_sync_slave; enable_warnings; connection master; set global rpl_semi_sync_master_enabled = 0; --source include/stop_dump_threads.inc disable_warnings; error 0,1305; UNINSTALL PLUGIN rpl_semi_sync_master; enable_warnings; connection slave; set global slave_compressed_protocol = 0; source include/start_slave.inc; connection master; --source include/rpl_end.inc ######################################## Testing this case in mysql-5.6.36 will be passed. but in mysql-5.7.18, we will get: mysql-test-run: WARNING: Got errors/warnings while running tests, please examine 'var/log/warnings' for details. in var/log/warnings: ... [ERROR] Read semi-sync reply magic number error [ERROR] Got timeout reading communication packets [ERROR] Got an error reading communication packets ... Suggested fix: plugin/semisync/semisync_master_ack_receiver.cc void Ack_receiver::run() { ... while (i < m_slaves.size()) { if (listener.is_socket_active(i)) { ulong len; net_clear(&net, 0); net.vio= &m_slaves[i].vio; /* set net.compress */ net.compress = *(&m_slaves[i].thd->net.compress); /* but now net is a private member of thd */ len= my_net_read(&net); if (likely(len != packet_error)) repl_semisync.reportReplyPacket(m_slaves[i].server_id(), net.read_pos, len); else if (net.last_errno == ER_NET_READ_ERROR) listener.clear_socket_info(i); } i++; } ... }