Bug #81205 In-place upgrade upgrade creates replication discrepancies
Submitted: 27 Apr 2016 4:35 Modified: 3 Apr 2018 23:14
Reporter: monty solomon Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:5.7.11,5.7.20,5.7.21 OS:CentOS
Assigned to: CPU Architecture:Any

[27 Apr 2016 4:35] monty solomon
Description:
After upgrading the slave and then the master from version 5.6 to 5.7 using the in-place upgrade there are replication discrepancies that were not present before the upgrade.

How to repeat:
Create new master and slave running MySQL 5.6.

Upgrade the slave to 5.7 using the in-place upgrade.

Upgrade the master to 5.7 using the in-place upgrade.

Compare the master and slave and observe multiple discrepancies.
[27 Apr 2016 4:37] monty solomon
Before the upgrade

pt-table-checksum --no-check-binlog-format --function murmur_hash -h empty-sun --recursion-method hosts

            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
04-27T04:09:35      0      0        1       1       0   0.173 heartbeat.heartbeat
04-27T04:09:35      0      0        0       1       0   0.036 mysql.columns_priv
04-27T04:09:35      0      0       17       1       0   0.035 mysql.db
04-27T04:09:35      0      0        0       1       0   0.037 mysql.event
04-27T04:09:35      0      0        3       1       0   0.035 mysql.func
04-27T04:09:35      0      0       40       1       0   0.034 mysql.help_category
04-27T04:09:35      0      0      485       1       0   0.035 mysql.help_keyword
04-27T04:09:35      0      0     1090       1       0   0.035 mysql.help_relation
04-27T04:09:35      0      0      533       1       0   0.035 mysql.help_topic
04-27T04:09:35      0      0        0       1       0   0.035 mysql.ndb_binlog_index
04-27T04:09:36      0      0        3       1       0   0.287 mysql.plugin
04-27T04:09:36      0      0        0       1       0   0.038 mysql.proc
04-27T04:09:36      0      0        0       1       0   0.035 mysql.procs_priv
04-27T04:09:36      0      0        6       1       0   0.035 mysql.proxies_priv
04-27T04:09:36      0      0        0       1       0   0.287 mysql.servers
04-27T04:09:36      0      0        0       1       0   0.035 mysql.tables_priv
04-27T04:09:36      0      0     1765       1       0   0.034 mysql.time_zone
04-27T04:09:36      0      0        0       1       0   0.034 mysql.time_zone_leap_second
04-27T04:09:36      0      0     1765       1       0   0.286 mysql.time_zone_name
04-27T04:09:37      0      0   118426       4       0   0.517 mysql.time_zone_transition
04-27T04:09:37      0      0     8296       1       0   0.290 mysql.time_zone_transition_type
04-27T04:09:37      0      0       24       1       0   0.036 mysql.user
[27 Apr 2016 4:37] monty solomon
After the slave upgrade

pt-table-checksum --no-check-binlog-format --function murmur_hash -h empty-sun --recursion-method hosts

            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
04-27T04:19:57      0      0        1       1       0   0.040 heartbeat.heartbeat
04-27T04:19:57      0      0        0       1       0   0.038 mysql.columns_priv
04-27T04:19:57      0      1       17       1       0   0.040 mysql.db
04-27T04:19:57      0      0        0       1       0   0.040 mysql.event
04-27T04:19:57      0      0        3       1       0   0.290 mysql.func
04-27T04:19:57      0      1       40       1       0   0.039 mysql.help_category
04-27T04:19:57      0      1      485       1       0   0.040 mysql.help_keyword
04-27T04:19:58      0      1     1090       1       0   0.038 mysql.help_relation
04-27T04:19:58      0      1      533       1       0   0.038 mysql.help_topic
04-27T04:19:58      0      0        0       1       0   0.037 mysql.ndb_binlog_index
04-27T04:19:58      0      0        3       1       0   0.290 mysql.plugin
04-27T04:19:58      0      1        0       1       0   0.038 mysql.proc
04-27T04:19:58      0      0        0       1       0   0.037 mysql.procs_priv
04-27T04:19:58      0      0        6       1       0   0.036 mysql.proxies_priv
04-27T04:19:58      0      0        0       1       0   0.037 mysql.servers
04-27T04:19:58      0      1        0       1       0   0.037 mysql.tables_priv
04-27T04:19:58      0      0     1765       1       0   0.291 mysql.time_zone
04-27T04:19:59      0      0        0       1       0   0.037 mysql.time_zone_leap_second
04-27T04:19:59      0      0     1765       1       0   0.037 mysql.time_zone_name
04-27T04:19:59      0      0   118426       4       0   0.520 mysql.time_zone_transition
04-27T04:19:59      0      0     8296       1       0   0.295 mysql.time_zone_transition_type
04-27T04:19:59 Skipping table mysql.user because it has problems on these replicas:
Table mysql.user on replica bold-band is missing these columns: password
This can break replication.  If you understand the risks, specify --no-check-slave-tables to disable this check.
[27 Apr 2016 4:38] monty solomon
After the master upgrade

pt-table-checksum --no-check-binlog-format --function murmur_hash -h empty-sun --recursion-method hosts

            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
04-27T04:26:32      0      0        1       1       0   0.291 heartbeat.heartbeat
04-27T04:26:32      0      0        0       1       0   0.039 mysql.columns_priv
04-27T04:26:32      0      0       18       1       0   0.038 mysql.db
04-27T04:26:32      0      1        2       1       0   0.038 mysql.engine_cost
04-27T04:26:32      0      0        0       1       0   0.038 mysql.event
04-27T04:26:32      0      0        3       1       0   0.037 mysql.func
04-27T04:26:32      0      0        0       1       0   0.289 mysql.gtid_executed
04-27T04:26:32      0      1       40       1       0   0.038 mysql.help_category
04-27T04:26:32      0      1      485       1       0   0.040 mysql.help_keyword
04-27T04:26:32      0      1     1090       1       0   0.039 mysql.help_relation
04-27T04:26:33      0      1      533       1       0   0.052 mysql.help_topic
04-27T04:26:33      0      0        0       1       0   0.037 mysql.ndb_binlog_index
04-27T04:26:33      0      0        3       1       0   0.038 mysql.plugin
04-27T04:26:33      0      1       47       1       0   0.038 mysql.proc
04-27T04:26:33      0      0        0       1       0   0.039 mysql.procs_priv
04-27T04:26:33      0      0        6       1       0   0.037 mysql.proxies_priv
04-27T04:26:33      0      1        6       1       0   0.037 mysql.server_cost
04-27T04:26:33      0      0        0       1       0   0.036 mysql.servers
04-27T04:26:33      0      1        1       1       0   0.038 mysql.tables_priv
04-27T04:26:33      0      0     1765       1       0   0.040 mysql.time_zone
04-27T04:26:33      0      0        0       1       0   0.037 mysql.time_zone_leap_second
04-27T04:26:33      0      0     1765       1       0   0.044 mysql.time_zone_name
04-27T04:26:34      0      0   118426       4       0   0.527 mysql.time_zone_transition
04-27T04:26:34      0      0     8296       1       0   0.300 mysql.time_zone_transition_type
04-27T04:26:34      0      1       25       1       0   0.041 mysql.user
04-27T04:26:34      0      1        6       1       0   0.042 sys.sys_config
[27 Apr 2016 4:43] monty solomon
Many of the discrepancies are timestamps. For example,

< INSERT INTO `engine_cost` VALUES ('default',0,'io_block_read_cost',NULL,'2016-04-27 04:24:42',NULL);
< INSERT INTO `engine_cost` VALUES ('default',0,'memory_block_read_cost',NULL,'2016-04-27 04:24:42',NULL);
---
> INSERT INTO `engine_cost` VALUES ('default',0,'io_block_read_cost',NULL,'2016-04-27 04:17:29',NULL);
> INSERT INTO `engine_cost` VALUES ('default',0,'memory_block_read_cost',NULL,'2016-04-27 04:17:29',NULL);
[27 Apr 2016 4:43] monty solomon
< INSERT INTO `server_cost` VALUES ('disk_temptable_create_cost',NULL,'2016-04-27 04:24:42',NULL);
< INSERT INTO `server_cost` VALUES ('disk_temptable_row_cost',NULL,'2016-04-27 04:24:42',NULL);
< INSERT INTO `server_cost` VALUES ('key_compare_cost',NULL,'2016-04-27 04:24:42',NULL);
< INSERT INTO `server_cost` VALUES ('memory_temptable_create_cost',NULL,'2016-04-27 04:24:42',NULL);
< INSERT INTO `server_cost` VALUES ('memory_temptable_row_cost',NULL,'2016-04-27 04:24:42',NULL);
< INSERT INTO `server_cost` VALUES ('row_evaluate_cost',NULL,'2016-04-27 04:24:42',NULL);
---
> INSERT INTO `server_cost` VALUES ('disk_temptable_create_cost',NULL,'2016-04-27 04:17:29',NULL);
> INSERT INTO `server_cost` VALUES ('disk_temptable_row_cost',NULL,'2016-04-27 04:17:29',NULL);
> INSERT INTO `server_cost` VALUES ('key_compare_cost',NULL,'2016-04-27 04:17:29',NULL);
> INSERT INTO `server_cost` VALUES ('memory_temptable_create_cost',NULL,'2016-04-27 04:17:29',NULL);
> INSERT INTO `server_cost` VALUES ('memory_temptable_row_cost',NULL,'2016-04-27 04:17:29',NULL);
> INSERT INTO `server_cost` VALUES ('row_evaluate_cost',NULL,'2016-04-27 04:17:29',NULL);
[27 Apr 2016 4:44] monty solomon
< INSERT INTO `tables_priv` VALUES ('localhost','sys','mysql.sys','sys_config','root@localhost','2016-04-27 04:24:43','Select','');
---
> INSERT INTO `tables_priv` VALUES ('localhost','sys','mysql.sys','sys_config','root@localhost','2016-04-27 04:17:31','Select','');
[28 Apr 2016 3:14] monty solomon
I tried using the command

mysql_upgrade --verbose --force --write-binlog 

with row binlog format and that caused a replication error.

               Last_SQL_Error: Could not execute Write_rows event on table mysql.server_cost; Duplicate entry 'row_evaluate_cost' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log bin.000003, end_log_pos 26986
[28 Nov 2017 17:35] MySQL Verification Team
Hi!

Upgrade of the slave and / or master while replication is running  is not recommended.
On the master, an upgrade should be done with binary log disabled, which is already enforced.
On the slave, mysql-upgrade never enables replication, so that in-place scenario with replication running can not happen Hence, the discrepancy that is noted after upgrade is expected.

You can read more about it in our manual in the chapter 16.1.3, mostly 16.1.3.4, also some in 2.11.1.

To do in-place upgrade, you have to stop replication, first upgrade slave, then master and then restart the replication. If you feel that it is not adequately described in chapter 16.1.3  or 2.11.1 , please let us know and we shall update our manual. What I wrote above is applicable to async replication only, not to Group Replication.

Needless to say, GTID_MODE and RELAY_LOG_RECOVERY should be both ON.

Please, also read:

https://bugs.mysql.com/bug.php?id=81840
[28 Nov 2017 22:57] monty solomon
The issue is not related to replication being running. The issue is that when mysql_upgrade runs it makes updates to tables that include timestamps related to when mysql_upgrade was run. It apparently makes updates to other tables too so that when it is run on the master later there are differences between the tables on the master and slaves.

The following tables had differences between the master and the slave after mysql_upgrade was run on the slave and then run on the master.

mysql.engine_cost
mysql.help_category
mysql.help_keyword
mysql.help_relation
mysql.help_topic
mysql.proc
mysql.server_cost
mysql.tables_priv
mysql.user
sys.sys_config

After running mysql_upgrade on the slave and then on the master the contents of some of the tables don't agree because of the timestamps that were updated.

For example, in the `mysql.engine_cost` table the `last_update` column is defined with DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP. When mysql_upgrade runs on the slave that field will get the current timestamp. When mysql_upgrade runs on the master later, that field will get a different timestamp. Then there is a replication discrepancy between the master and the slave for the `engine_cost` table .

There are similar issues with the `mysql.server_cost` table
  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

and other tables

`sys_config`
  `set_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

`mysql.tables_priv`
  `Timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

`mysql.proc
   `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
[29 Nov 2017 13:01] MySQL Verification Team
Hi!

In my response to you, I have provided a location of one chapter in our manual, dealing with the problems that may arise when mysql_upgrade is used with --write-binlog and when one of the server options is setup. 

Have you checked that possibility ???
[29 Nov 2017 16:01] MySQL Verification Team
In my opinion the real complaint here is that 5.7 has a non-deterministic initial dataset because it populates the current system timestamp into a few tables/fields.

Creating two new servers one after the other with mysqld --initialize-insecure will lead to a different database content each time.

See:
 https://pastebin.com/raw/Kc8iaJqd
[30 Nov 2017 7:22] monty solomon
I didn't run mysql_upgrade with --write-binlog during the process that created the discrepancies between the master and the slave. I tried to use the --write-binlog option to fix the discrepancies after I discovered them.

The problem I described is independent of using GTID. When mysql_upgrade runs and creates new tables or modifies existing tables it can cause timestamps to be created or updated at the time it was run. Since mysql_upgrade is run separately on the slave and on the master those timestamps will be different.
[30 Nov 2017 7:51] monty solomon
The sections you referenced need to be updated. Section 2.11.1 states that mysql_upgrade should not be used with GTID but sections 16.1.3.4 and 4.4.7 state that only --write-binlog should not be used with GTID.

Sections 2.11.1 states

mysql_upgrade should not be used when the server is running with --gtid-mode=ON. See GTID mode and mysql_upgrade for more information.

Section 16.1.3.4 states

GTID mode and mysql_upgrade.  It is not recommended to use mysql_upgrade with the --write-binlog option on a MySQL Server running with --gtid-mode=ON because mysql_upgrade can make changes to system tables that use the MyISAM storage engine, which is non-transactional.

Section 4.4.7 states with respect to using --write-binlog

this option is not recommended on MySQL Servers running with --gtid-mode=ON because mysql_upgrade can make changes to system tables that use the MyISAM storage engine, which is non-transactional.
[30 Nov 2017 14:00] MySQL Verification Team
Hi Monty,

First and first of all, that bug that you wrote about has been open and duly re-assigned.

Second, let us first fix the documentation, which is why I am verifying this as a documentation bug.

Meanwhile, please try to make a minimal case where mysql_upgrade will change timestamps in the way that you described. Then, after we manage to repeat the behaviour, I will change this bug into a code bug, related to mysql_upgrade.

Couple of questions related to the above. First, in 5.6 did you have temporal values already converted to the new format or did you still hold them in the old, 5.5 format ???

Second question is about the conditions. Does mysql_upgrade makes a mess that you describe on standalone server too, or only on the slave ???

Last, but not least, if you wish you can open a new bug that will deal with the problems caused by some code bug , when you have a test case.  Or we can continue with this bug. As you wish .....

Many thanks in advance.
[6 Dec 2017 5:15] monty solomon
1. Disable the 5.7 repo and install mysql-community-server-5.6.38 on a master and two slaves

yum install mysql-community-server
mysql_install_db --user=mysql
service mysqld start

2. Dump the mysql database from slave one

mysqldump --single-transaction --databases mysql -r mysql-56.sql --skip-extended-insert

3. Enable the 5.7 repo and upgrade to mysql-community-server-5.7.20 on slave one

yum upgrade mysql-community-server*

4. Run mysql_upgrade on slave one and restart

mysql_upgrade -uroot --verbose --force --skip-write-binlog
service mysqld restart

5. Dump the mysql database from slave one

mysqldump --single-transaction --databases mysql -r mysql-57.sql --skip-extended-insert

6. Compare the dumps from steps 2 and 5.

diff mysql-5* | grep "2017-12" | grep INSERT

7. Observe that many of the entries added by mysql_upgrade to multiple tables have a timestamp.

When the same process is repeated on slave two to upgrade it to 5.7, the added entries will have different timestamps from those on slave one.

When the same process is repeated on the master to upgrade it to 5.7, the added entries will have different timestamps from those on both of the slaves.

There are now replication discrepancies between tables in the mysql database on the master and the slaves.

Here are some of the INSERT statements with timestamps from the diff (I didn't include the entries from the `proc` table):

> INSERT INTO `engine_cost` VALUES ('default',0,'io_block_read_cost',NULL,'2017-12-06 04:26:52',NULL);
> INSERT INTO `engine_cost` VALUES ('default',0,'memory_block_read_cost',NULL,'2017-12-06 04:26:52',NULL);

> INSERT INTO `innodb_index_stats` VALUES ('mysql','gtid_executed','PRIMARY','2017-12-06 04:26:51','n_diff_pfx01',0,1,'source_uuid');
> INSERT INTO `innodb_index_stats` VALUES ('mysql','gtid_executed','PRIMARY','2017-12-06 04:26:51','n_diff_pfx02',0,1,'source_uuid,interval_start');
> INSERT INTO `innodb_index_stats` VALUES ('mysql','gtid_executed','PRIMARY','2017-12-06 04:26:51','n_leaf_pages',1,NULL,'Number of leaf pages in the index');
> INSERT INTO `innodb_index_stats` VALUES ('mysql','gtid_executed','PRIMARY','2017-12-06 04:26:51','size',1,NULL,'Number of pages in the index');
> INSERT INTO `innodb_index_stats` VALUES ('sys','sys_config','PRIMARY','2017-12-06 04:27:02','n_diff_pfx01',6,1,'variable');
> INSERT INTO `innodb_index_stats` VALUES ('sys','sys_config','PRIMARY','2017-12-06 04:27:02','n_leaf_pages',1,NULL,'Number of leaf pages in the index');
> INSERT INTO `innodb_index_stats` VALUES ('sys','sys_config','PRIMARY','2017-12-06 04:27:02','size',1,NULL,'Number of pages in the index');
> INSERT INTO `innodb_table_stats` VALUES ('mysql','gtid_executed','2017-12-06 04:26:51',0,1,0);
> INSERT INTO `innodb_table_stats` VALUES ('sys','sys_config','2017-12-06 04:27:02',6,1,0);

> INSERT INTO `server_cost` VALUES ('disk_temptable_create_cost',NULL,'2017-12-06 04:26:51',NULL);
> INSERT INTO `server_cost` VALUES ('disk_temptable_row_cost',NULL,'2017-12-06 04:26:51',NULL);
> INSERT INTO `server_cost` VALUES ('key_compare_cost',NULL,'2017-12-06 04:26:51',NULL);
> INSERT INTO `server_cost` VALUES ('memory_temptable_create_cost',NULL,'2017-12-06 04:26:51',NULL);
> INSERT INTO `server_cost` VALUES ('memory_temptable_row_cost',NULL,'2017-12-06 04:26:51',NULL);
> INSERT INTO `server_cost` VALUES ('row_evaluate_cost',NULL,'2017-12-06 04:26:51',NULL);

> INSERT INTO `tables_priv` VALUES ('localhost','mysql','mysql.session','user','root@localhost','2017-12-06 04:26:52','Select','');
> INSERT INTO `tables_priv` VALUES ('localhost','sys','mysql.sys','sys_config','root@localhost','2017-12-06 04:26:52','Select','');

> INSERT INTO `user` VALUES ('localhost','root','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'mysql_native_password','','N','2017-12-06 04:26:52',NULL,'N');
> INSERT INTO `user` VALUES ('open-night','root','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'mysql_native_password','','N','2017-12-06 04:26:52',NULL,'N');
> INSERT INTO `user` VALUES ('127.0.0.1','root','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'mysql_native_password','','N','2017-12-06 04:26:52',NULL,'N');
> INSERT INTO `user` VALUES ('::1','root','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','Y','','','','',0,0,0,0,'mysql_native_password','','N','2017-12-06 04:26:52',NULL,'N');
> INSERT INTO `user` VALUES ('localhost','','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','','','','',0,0,0,0,'mysql_native_password',NULL,'N','2017-12-06 04:26:52',NULL,'N');
> INSERT INTO `user` VALUES ('open-night','','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','','','','',0,0,0,0,'mysql_native_password',NULL,'N','2017-12-06 04:26:52',NULL,'N');
> INSERT INTO `user` VALUES ('localhost','mysql.session','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','Y','N','N','N','N','N','N','N','N','N','N','N','N','N','','','','',0,0,0,0,'mysql_native_password','*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE','N','2017-12-06 04:26:52',NULL,'Y');
> INSERT INTO `user` VALUES ('localhost','mysql.sys','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','','','','',0,0,0,0,'mysql_native_password','*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE','N','2017-12-06 04:26:52',NULL,'Y');
[20 Dec 2017 11:26] Margaret Fisher
Posted by developer:
 
Documentation updated to remove references to the --write-binlog option at
https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-restrictions.html
https://dev.mysql.com/doc/refman/5.7/en/mysql-upgrade.html

The restriction applies to using mysql-upgrade with GTIDs at all, and is now stated as

It is not recommended to use mysql_upgrade on a MySQL Server running with --gtid-mode=ON, because mysql_upgrade can make changes to system tables that use the MyISAM storage engine, which is non-transactional.
[20 Dec 2017 20:31] monty solomon
Please add some detailed instructions to explain how to upgrade MySQL when using GTIDs.
[20 Dec 2017 20:32] monty solomon
What is the significance of the non-transactional tables? How are they related to GTIDs? When mysql_upgrade is run it doesn't write to the binary log and therefore doesn't create any GTIDs.
[3 Apr 2018 23:09] monty solomon
I upgraded from 5.7.18 to 5.7.21 and new replication discrepancies were created due to different timestamps at the time of upgrade.

mysql> select * from tables_priv where user='mysql.session'\G
*************************** 1. row ***************************
       Host: localhost
         Db: mysql
       User: mysql.session
 Table_name: user
    Grantor: root@localhost
  Timestamp: 2018-03-27 22:01:28
 Table_priv: Select
Column_priv: 
1 row in set (0.00 sec)

mysql> select * from tables_priv where user='mysql.session'\G
*************************** 1. row ***************************
       Host: localhost
         Db: mysql
       User: mysql.session
 Table_name: user
    Grantor: root@localhost
  Timestamp: 2018-03-27 20:01:12
 Table_priv: Select
Column_priv: 
1 row in set (0.00 sec)
[3 Apr 2018 23:10] monty solomon
The timestamp differs in the user table too

mysql> select * from user where User='mysql.session'\G
*************************** 1. row ***************************
                  Host: localhost
                  User: mysql.session
           Select_priv: N
           Insert_priv: N
           Update_priv: N
           Delete_priv: N
           Create_priv: N
             Drop_priv: N
           Reload_priv: N
         Shutdown_priv: N
          Process_priv: N
             File_priv: N
            Grant_priv: N
       References_priv: N
            Index_priv: N
            Alter_priv: N
          Show_db_priv: N
            Super_priv: Y
 Create_tmp_table_priv: N
      Lock_tables_priv: N
          Execute_priv: N
       Repl_slave_priv: N
      Repl_client_priv: N
      Create_view_priv: N
        Show_view_priv: N
   Create_routine_priv: N
    Alter_routine_priv: N
      Create_user_priv: N
            Event_priv: N
          Trigger_priv: N
Create_tablespace_priv: N
              ssl_type: 
            ssl_cipher: 
           x509_issuer: 
          x509_subject: 
         max_questions: 0
           max_updates: 0
       max_connections: 0
  max_user_connections: 0
                plugin: mysql_native_password
 authentication_string: *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE
      password_expired: N
 password_last_changed: 2018-03-27 20:01:12
     password_lifetime: NULL
        account_locked: Y
1 row in set (0.01 sec)

mysql> select * from user where User='mysql.session'\G
*************************** 1. row ***************************
                  Host: localhost
                  User: mysql.session
           Select_priv: N
           Insert_priv: N
           Update_priv: N
           Delete_priv: N
           Create_priv: N
             Drop_priv: N
           Reload_priv: N
         Shutdown_priv: N
          Process_priv: N
             File_priv: N
            Grant_priv: N
       References_priv: N
            Index_priv: N
            Alter_priv: N
          Show_db_priv: N
            Super_priv: Y
 Create_tmp_table_priv: N
      Lock_tables_priv: N
          Execute_priv: N
       Repl_slave_priv: N
      Repl_client_priv: N
      Create_view_priv: N
        Show_view_priv: N
   Create_routine_priv: N
    Alter_routine_priv: N
      Create_user_priv: N
            Event_priv: N
          Trigger_priv: N
Create_tablespace_priv: N
              ssl_type: 
            ssl_cipher: 
           x509_issuer: 
          x509_subject: 
         max_questions: 0
           max_updates: 0
       max_connections: 0
  max_user_connections: 0
                plugin: mysql_native_password
 authentication_string: *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE
      password_expired: N
 password_last_changed: 2018-03-27 22:01:28
     password_lifetime: NULL
        account_locked: Y
1 row in set (0.00 sec)
[3 Apr 2018 23:14] monty solomon
Output from pt-table-checksum

            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
04-02T03:47:11      0      1        7       1       0   0.309 mysql.tables_priv
04-02T03:47:12      0      1       44       1       0   0.060 mysql.user

TABLE CHUNK CNT_DIFF CRC_DIFF CHUNK_INDEX LOWER_BOUNDARY UPPER_BOUNDARY
mysql.tables_priv 1 0 1   
mysql.user 1 0 1
[4 Apr 2018 4:53] MySQL Verification Team
Pasting my older comment:

In my opinion the real complaint here is that 5.7 has a non-deterministic initial dataset because it populates the current system timestamp into a few tables/fields.

Creating two new servers one after the other with mysqld --initialize-insecure will lead to a different database content each time.

See:
 https://pastebin.com/raw/Kc8iaJqd
[4 Apr 2018 4:54] MySQL Verification Team
this has nothing to do with replication