MySQL Bugs: #91484: mysqlrpladmin gitd and health commands don't report errant or missing GTIDs

Bug #91484	mysqlrpladmin gitd and health commands don't report errant or missing GTIDs
Submitted:	29 Jun 2018 5:25	Modified:	2 Sep 2018 0:20
Reporter:	monty solomon	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Utilities	Severity:	S3 (Non-critical)
Version:		OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
I run mysqlrpladmin gitd and mysqlrpladmin health to check the status of the cluster before running mysqlrpladmin switchover. mysqlrpladmin switchover fails if there is a problem with the GTIDs.

I expected mysqlrpladmin gitd or mysqlrpladmin health to find and report any problems before running mysqlrpladmin switchover.

How to repeat:
Execute a statement on the slave that adds an errant GTID.

Run mysqlrpladmin gitd

Run mysqlrpladmin health

Run mysqlrpladmin switchover

Observe failure message.

Suggested fix:
Detect and report any GTID problems in mysqlrpladmin gitd and mysqlrpladmin health.

mysqlrpladmin gtid --verbose --discover-slaves-login=rpladmin --master=zany-lion --log=mysqlrpladmin.log --timeout=10

NOTE: Log file 'mysqlrpladmin.log' does not exist. Will be created., 
# Discovering slaves for master at zany-lion:3306, 
# Discovering slave at brief-spruce:3306, 
# Found slave: brief-spruce:3306, 
# Discovering slave at plain-turtle:3306, 
# Found slave: plain-turtle:3306, 
# Checking privileges., 
#, 
# UUIDS for all servers:, 
+------------------------------------------+-------+---------+---------------------------------------+, 
| host          | port  | role    | uuid       |, 
+------------------------------------------+-------+---------+---------------------------------------+, 
| zany-lion     | 3306  | MASTER  | 67e98ae6-5d46-11e8-92d0-0e2ea7677c82  |, 
| brief-spruce  | 3306  | SLAVE   | d8abfea8-5d4b-11e8-a21c-0a7d8b616102  |, 
| plain-turtle  | 3306  | SLAVE   | b1187e3f-5d4f-11e8-b070-1235daf25e4a  |, 
+------------------------------------------+-------+---------+---------------------------------------+, 
#, 
# Transactions executed on the server:, 
+------------------------------------------+-------+---------+-------------------------------------------------+, 
| host          | port  | role    | gtid                 |, 
+------------------------------------------+-------+---------+-------------------------------------------------+, 
| zany-lion     | 3306  | MASTER  | 0fa5b8f5-1291-11e8-9f75-0ee20b4e91dc:1-77939    |, 
| zany-lion     | 3306  | MASTER  | 40edc397-ad30-11e7-94c7-1273125712b4:1-9574655  |, 
| zany-lion     | 3306  | MASTER  | 493b37e9-3624-11e8-afa8-12ccf2c3b916:1-1043628  |, 
| zany-lion     | 3306  | MASTER  | 58b61db7-fc96-11e7-ac0b-0ee20b4e91dc:1-2189566  |, 
| zany-lion     | 3306  | MASTER  | 67e98ae6-5d46-11e8-92d0-0e2ea7677c82:1-3303386  |, 
| zany-lion     | 3306  | MASTER  | e5df10c4-c96f-11e7-baa5-0ee20b4e91dc:1-22       |, 
| zany-lion     | 3306  | MASTER  | f6211e73-ef78-11e7-aeb5-0ee20b4e91dc:1-1501     |, 
| brief-spruce  | 3306  | SLAVE   | 0fa5b8f5-1291-11e8-9f75-0ee20b4e91dc:1-77939    |, 
| brief-spruce  | 3306  | SLAVE   | 40edc397-ad30-11e7-94c7-1273125712b4:1-9574655  |, 
| brief-spruce  | 3306  | SLAVE   | 493b37e9-3624-11e8-afa8-12ccf2c3b916:1-1043628  |, 
| brief-spruce  | 3306  | SLAVE   | 58b61db7-fc96-11e7-ac0b-0ee20b4e91dc:1-2189566  |, 
| brief-spruce  | 3306  | SLAVE   | 67e98ae6-5d46-11e8-92d0-0e2ea7677c82:1-3303386  |, 
| brief-spruce  | 3306  | SLAVE   | d8abfea8-5d4b-11e8-a21c-0a7d8b616102:1          |, 
| brief-spruce  | 3306  | SLAVE   | e5df10c4-c96f-11e7-baa5-0ee20b4e91dc:1-22       |, 
| brief-spruce  | 3306  | SLAVE   | f6211e73-ef78-11e7-aeb5-0ee20b4e91dc:1-1501     |, 
| plain-turtle  | 3306  | SLAVE   | 0fa5b8f5-1291-11e8-9f75-0ee20b4e91dc:1-77939    |, 
| plain-turtle  | 3306  | SLAVE   | 40edc397-ad30-11e7-94c7-1273125712b4:1-9574655  |, 
| plain-turtle  | 3306  | SLAVE   | 493b37e9-3624-11e8-afa8-12ccf2c3b916:1-1043628  |, 
| plain-turtle  | 3306  | SLAVE   | 58b61db7-fc96-11e7-ac0b-0ee20b4e91dc:1-2189566  |, 
| plain-turtle  | 3306  | SLAVE   | 67e98ae6-5d46-11e8-92d0-0e2ea7677c82:1-3303386  |, 
| plain-turtle  | 3306  | SLAVE   | e5df10c4-c96f-11e7-baa5-0ee20b4e91dc:1-22       |, 
| plain-turtle  | 3306  | SLAVE   | f6211e73-ef78-11e7-aeb5-0ee20b4e91dc:1-1501     |, 
+------------------------------------------+-------+---------+-------------------------------------------------+, 
#, 
# Transactions purged from the server:, 
+------------------------------------------+-------+---------+-------------------------------------------------+, 
| host          | port  | role    | gtid                 |, 
+------------------------------------------+-------+---------+-------------------------------------------------+, 
| zany-lion     | 3306  | MASTER  | 0fa5b8f5-1291-11e8-9f75-0ee20b4e91dc:1-77939    |, 
| zany-lion     | 3306  | MASTER  | 40edc397-ad30-11e7-94c7-1273125712b4:1-9574655  |, 
| zany-lion     | 3306  | MASTER  | 493b37e9-3624-11e8-afa8-12ccf2c3b916:1-1043628  |, 
| zany-lion     | 3306  | MASTER  | 58b61db7-fc96-11e7-ac0b-0ee20b4e91dc:1-2189566  |, 
| zany-lion     | 3306  | MASTER  | 67e98ae6-5d46-11e8-92d0-0e2ea7677c82:1-3128290  |, 
| zany-lion     | 3306  | MASTER  | e5df10c4-c96f-11e7-baa5-0ee20b4e91dc:1-22       |, 
| zany-lion     | 3306  | MASTER  | f6211e73-ef78-11e7-aeb5-0ee20b4e91dc:1-1501     |, 
| brief-spruce  | 3306  | SLAVE   | 0fa5b8f5-1291-11e8-9f75-0ee20b4e91dc:1-77939    |, 
| brief-spruce  | 3306  | SLAVE   | 40edc397-ad30-11e7-94c7-1273125712b4:1-9574655  |, 
| brief-spruce  | 3306  | SLAVE   | 493b37e9-3624-11e8-afa8-12ccf2c3b916:1-1043628  |, 
| brief-spruce  | 3306  | SLAVE   | 58b61db7-fc96-11e7-ac0b-0ee20b4e91dc:1-2189566  |, 
| brief-spruce  | 3306  | SLAVE   | 67e98ae6-5d46-11e8-92d0-0e2ea7677c82:1-3128553  |, 
| brief-spruce  | 3306  | SLAVE   | e5df10c4-c96f-11e7-baa5-0ee20b4e91dc:1-22       |, 
| brief-spruce  | 3306  | SLAVE   | f6211e73-ef78-11e7-aeb5-0ee20b4e91dc:1-1501     |, 
| plain-turtle  | 3306  | SLAVE   | 0fa5b8f5-1291-11e8-9f75-0ee20b4e91dc:1-77939    |, 
| plain-turtle  | 3306  | SLAVE   | 40edc397-ad30-11e7-94c7-1273125712b4:1-9574655  |, 
| plain-turtle  | 3306  | SLAVE   | 493b37e9-3624-11e8-afa8-12ccf2c3b916:1-1043628  |, 
| plain-turtle  | 3306  | SLAVE   | 58b61db7-fc96-11e7-ac0b-0ee20b4e91dc:1-2189566  |, 
| plain-turtle  | 3306  | SLAVE   | 67e98ae6-5d46-11e8-92d0-0e2ea7677c82:1-3128272  |, 
| plain-turtle  | 3306  | SLAVE   | e5df10c4-c96f-11e7-baa5-0ee20b4e91dc:1-22       |, 
| plain-turtle  | 3306  | SLAVE   | f6211e73-ef78-11e7-aeb5-0ee20b4e91dc:1-1501     |, 
+------------------------------------------+-------+---------+-------------------------------------------------+, 
# ...done.

mysqlrpladmin health --verbose --discover-slaves-login=rpladmin --master=zany-lion --log=mysqlrpladmin.log --timeout=10

# Discovering slaves for master at zany-lion:3306, 
# Discovering slave at brief-spruce:3306, 
# Found slave: brief-spruce:3306, 
# Discovering slave at plain-turtle:3306, 
# Found slave: plain-turtle:3306, 
# Checking privileges., 
# Attempting to contact zany-lion ... Success, 
# Attempting to contact brief-spruce ... Success, 
# Attempting to contact plain-turtle ... Success, 
#, 
# Replication Topology Health:, 
+------------------------------------------+-------+---------+--------+------------+---------+----------------+------------------+-----------------+------------+-------------+--------------+------------------+---------------+-----------+----------------+------------+---------------+, 
| host          | port  | role    | state  | gtid_mode  | health  | version        | master_log_file  | master_log_pos  | IO_Thread  | SQL_Thread  | Secs_Behind  | Remaining_Delay  | IO_Error_Num  | IO_Error  | SQL_Error_Num  | SQL_Error  | Trans_Behind  |, 
+------------------------------------------+-------+---------+--------+------------+---------+----------------+------------------+-----------------+------------+-------------+--------------+------------------+---------------+-----------+----------------+------------+---------------+, 
| zany-lion     | 3306  | MASTER  | UP     | ON         | OK      | 5.7.21-21-log  | bin.003047       | 161309          |            |             |              |                  |               |           |                |            |               |, 
| brief-spruce  | 3306  | SLAVE   | UP     | ON         | OK      | 5.7.21-21-log  | bin.003047       | 161309          | Yes        | Yes         | 0            | No               | 0             |           | 0              |            | 0             |, 
| plain-turtle  | 3306  | SLAVE   | UP     | ON         | OK      | 5.7.21-21-log  | bin.003047       | 161688          | Yes        | Yes         | 0            | No               | 0             |           | 0              |            | 0             |, 
+------------------------------------------+-------+---------+--------+------------+---------+----------------+------------------+-----------------+------------+-------------+--------------+------------------+---------------+-----------+----------------+------------+---------------+, 
# ...done.

mysqlrpladmin switchover --verbose --discover-slaves-login=rpladmin --master=zany-lion --new-master=brief-spruce --demote-master --rpl-user=repl --log=mysqlrpladmin.log --timeout=240

# Discovering slaves for master at zany-lion:3306
# Discovering slave at brief-spruce:3306
# Found slave: brief-spruce:3306
# Discovering slave at plain-turtle:3306
# Found slave: plain-turtle:3306
# Checking privileges.
# ERROR: Errant transaction(s) found on slave(s).
#  - For slave 'brief-spruce@3306': d8abfea8-5d4b-11e8-a21c-0a7d8b616102:1
ERROR: Errant transaction(s) found on slave(s). Note: If you want to ignore this issue, although not advised, please use the utility with the --force option.

Hi,
Thanks for the report. I verified it. Looks like full check is only done on switchover and not before.

thanks for report
Bogdan