Bug #89533 MTR test rpl_gtid.rpl_multi_source_block_receiver failing testcase check
Submitted: 5 Feb 2018 12:51 Modified: 26 Mar 2018 14:25
Reporter: Laurynas Biveinis (OCA) Email Updates:
Status: Verified Impact on me:
None 
Category:Tests: Replication Severity:S3 (Non-critical)
Version:8.0.4 OS:Any
Assigned to: CPU Architecture:Any
Triage: Needs Triage: D5 (Feature request)

[5 Feb 2018 12:51] Laurynas Biveinis
Description:
Intermittently:

rpl_gtid.rpl_multi_source_block_receiver 'stmt' w1 [ fail ]
        Test ended at 2018-02-05 01:34:28

MTR's internal check of the test case 'rpl_gtid.rpl_multi_source_block_receiver' failed.
This means that the test case does not preserve the state that existed
before the test case was executed.  Most likely the test case did not
do a proper clean-up. It could also be caused by the previous test run
by this thread, if the server wasn't restarted.
This is the diff of the states of the servers before and after the
test case was executed:
mysqltest: Results saved in '/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/min-centos-7-x64/build/mysql-test/var/1/tmp/check-mysqld_2.result'.
mysqltest: Logging to '/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/min-centos-7-x64/build/mysql-test/var/1/tmp/check-mysqld_2.log'.
mysqltest: Connecting to server localhost:13001 (socket /tmp/2cBaIuT0W1/1/mysqld.2.sock) as 'root', connection 'default', attempt 0 ...
mysqltest: ... Connected.
mysqltest: Start processing test commands from './include/check-testcase.test' ...
mysqltest: ... Done processing test commands.
--- /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/min-centos-7-x64/build/mysql-test/var/1/tmp/check-mysqld_2.result	2018-02-05 09:34:24.547744599 +0300
+++ /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/min-centos-7-x64/build/mysql-test/var/1/tmp/check-mysqld_2.reject	2018-02-05 09:34:28.234790310 +0300
@@ -633,6 +633,7 @@
 def	test	utf8mb4	utf8mb4_0900_ai_ci	NULL
 FILE_ID	FILE_NAME	FILE_TYPE	TABLESPACE_NAME	TABLE_CATALOG	TABLE_SCHEMA	TABLE_NAME	LOGFILE_GROUP_NAME	LOGFILE_GROUP_NUMBER	ENGINE	FULLTEXT_KEYS	DELETED_ROWS	UPDATE_COUNT	FREE_EXTENTS	TOTAL_EXTENTS	EXTENT_SIZE	INITIAL_SIZE	MAXIMUM_SIZE	AUTOEXTEND_SIZE	CREATION_TIME	LAST_UPDATE_TIME	LAST_ACCESS_TIME	RECOVER_TIME	TRANSACTION_COUNTER	VERSION	ROW_FORMAT	TABLE_ROWS	AVG_ROW_LENGTH	DATA_LENGTH	MAX_DATA_LENGTH	INDEX_LENGTH	DATA_FREE	CREATE_TIME	UPDATE_TIME	CHECK_TIME	CHECKSUM	STATUS	EXTRA
 tables_in_test
+t2
 tables_in_mysql
 mysql.columns_priv
 mysql.component

mysqltest: Result content mismatch

How to repeat:
Not sure, probably likely to happen under load

Suggested fix:
The relevant testcase snippet is

--let $rpl_connection_name= server_3
--source include/rpl_connection.inc
DROP TABLE t2;

--let $rpl_connection_name= server_2
--source include/rpl_connection.inc
--let $rpl_channel_name= 'channel_3'
--source include/start_slave.inc

--let $rpl_skip_sync=1
--source include/rpl_end.inc

It looks like the issue is missing sync on channel_3 after starting the slave there.
[23 Mar 2018 8:55] Umesh Shastry
Hello Laurynas,

Thank you for the report and feedback.
I tried reproduce this on almost all available instances(OL7, Ubuntu, CentOS7 etc with src debug builds - mysql-boost-8.0.4-rc.tar.gz) but not seeing any issues. Could you please provide exact cmake with the options used? Thanks!

Regards,
Umesh
[23 Mar 2018 13:14] Umesh Shastry
##
ushastry@ArtfulAardvark:~/Downloads$ cmake --version
cmake version 3.9.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).
ushastry@ArtfulAardvark:~/Downloads$ gcc --version
gcc (Ubuntu 7.2.0-8ubuntu3.2) 7.2.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

ushastry@ArtfulAardvark:~/Downloads$ uname -aa
Linux ArtfulAardvark 4.13.0-36-generic #40-Ubuntu SMP Fri Feb 16 20:07:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
ushastry@ArtfulAardvark:~/Downloads$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.10
DISTRIB_CODENAME=artful
DISTRIB_DESCRIPTION="Ubuntu 17.10"
NAME="Ubuntu"
VERSION="17.10 (Artful Aardvark)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.10"
VERSION_ID="17.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=artful
UBUNTU_CODENAME=artful

ushastry@ArtfulAardvark:~/Downloads/mysql-8.0.4-rc$ cmake .  -DDOWNLOAD_BOOST=1 -DWITH_BOOST=./boost -DWITH_ASAN=ON -DWITH_UBSAN=ON -DWITH_DEBUG=1 -DCMAKE_INSTALL_PREFIX=/home/ushastry/Downloads/8_4_ASAN_UBSAN
ushastry@ArtfulAardvark:~/Downloads/mysql-8.0.4-rc$ make -j8
ushastry@ArtfulAardvark:~/Downloads/mysql-8.0.4-rc$ make install

ushastry@ArtfulAardvark:~/Downloads$ cd 8_4_ASAN_UBSAN/
ushastry@ArtfulAardvark:~/Downloads/8_4_ASAN_UBSAN$ cd mysql-test/
ushastry@ArtfulAardvark:~/Downloads/8_4_ASAN_UBSAN/mysql-test$ cd ..
ushastry@ArtfulAardvark:~/Downloads/8_4_ASAN_UBSAN$ cat docs/INFO_SRC
commit: 73b6cff33896a3cd0f0d02d6c30fdaed8b562cf3
date: 2018-01-18 16:43:56 +0100
build-date: 2018-01-18 16:53:08 +0100
short: 73b6cff
branch: mysql-8.0.4-rc-release

MySQL source 8.0.4
ushastry@ArtfulAardvark:~/Downloads/8_4_ASAN_UBSAN$ cd mysql-test/

ushastry@ArtfulAardvark:~/Downloads/8_4_ASAN_UBSAN/mysql-test$ ./mtr --debug-server rpl_gtid.rpl_multi_source_block_receiver
Logging: ./mtr  --debug-server rpl_gtid.rpl_multi_source_block_receiver
2018-03-23T13:00:49.423467Z 0 [System] [MY-010116] /home/ushastry/Downloads/8_4_ASAN_UBSAN/bin/mysqld (mysqld 8.0.4-rc-debug-log-asan-ubsan) starting as process 20047 ...
MySQL Version 8.0.4
Checking supported features...
 - SSL connections supported
 - binaries are debug compiled
Collecting tests...
 - adding combinations for rpl_gtid
Checking leftover processes...
Removing old var directory...
Creating var directory '/home/ushastry/Downloads/8_4_ASAN_UBSAN/mysql-test/var'...
Installing system database...
Using parallel: 1

==============================================================================

TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009
rpl_gtid.rpl_multi_source_block_receiver 'mix' [ pass ]  10379
rpl_gtid.rpl_multi_source_block_receiver 'row' [ pass ]  11090
rpl_gtid.rpl_multi_source_block_receiver 'stmt' [ pass ]  10672
--------------------------------------------------------------------------
The servers were restarted 2 times
Spent 32.141 of 228 seconds executing testcases

Completed: All 3 tests were successful.
[26 Mar 2018 7:57] Laurynas Biveinis
Umesh -

I have lost the exact CMake invocation for this particular reproduction. In any case this should be load-, not cmake-related, and I have suggested an IMHO very plausible cause and fix in the bug report, it could be handled as a testcase improvement request then.
[26 Mar 2018 14:25] Umesh Shastry
Thank you for the feedback.
Verifying as a testcase improvement request.

Regards,
Umesh