MySQL Bugs: #111978: Our PRD env mysql 8.0.30 group-replication cluster lost some data

Bug #111978	Our PRD env mysql 8.0.30 group-replication cluster lost some data
Submitted:	4 Aug 2023 14:22	Modified:	10 Aug 2023 14:27
Reporter:	Yuanjun Li	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Cluster: Replication	Severity:	S1 (Critical)
Version:	8.0.30	OS:	CentOS (CentOS Linux release 7.7.1908 (Core))
Assigned to:	MySQL Verification Team	CPU Architecture:	x86 (Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz)

Description:
Here is a insert binlog in table t_after_services with primary key 16820 on 2023/07/19 9:50:13:

#230719  9:50:13 server id 32  end_log_pos 580621239 	GTID	last_committed=99455	sequence_number=99456	rbr_only=yes	original_committed_timestamp=1689731413730952	immediate_commit_timestamp=1689731413731853	transaction_length=2041
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
# original_commit_timestamp=1689731413730952 (2023-07-19 09:50:13.730952 CST)
# immediate_commit_timestamp=1689731413731853 (2023-07-19 09:50:13.731853 CST)
/*!80001 SET @@session.original_commit_timestamp=1689731413730952*//*!*/;
/*!80014 SET @@session.original_server_version=80030*//*!*/;
/*!80014 SET @@session.immediate_server_version=80030*//*!*/;
SET @@SESSION.GTID_NEXT= 'cb1e163d-94e7-47e2-880f-82a481908bcf:67276184'/*!*/;
# at 580621239
#230719  9:50:13 server id 32  end_log_pos 580621318 	Query	thread_id=1089778	exec_time=0	error_code=0
SET TIMESTAMP=1689731413/*!*/;
BEGIN
/*!*/;
# at 580621318
#230719  9:50:13 server id 32  end_log_pos 580621499 	Table_map: `small_oa_107`.`t_after_services` mapped to number 6956330
# has_generated_invisible_primary_key=0
# at 580621499
#230719  9:50:13 server id 32  end_log_pos 580621758 	Write_rows: table id 6956330 flags: STMT_END_F
### INSERT INTO `small_oa_107`.`t_after_services`
### SET
###   @1=16820 /* LONGINT meta=0 nullable=0 is_null=0 */
###   @2=89491 /* LONGINT meta=0 nullable=0 is_null=0 */
###   @3='139911722xx' /* VARSTRING(44) meta=44 nullable=1 is_null=0 */
###   @4='先生' /* VARSTRING(200) meta=200 nullable=1 is_null=0 */
###   @5=0 /* TINYINT meta=0 nullable=0 is_null=0 */
###   @6=NULL /* TIMESTAMP(0) meta=0 nullable=1 is_null=1 */
###   @7=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @8=19 /* LONGINT meta=0 nullable=0 is_null=0 */
###   @9=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @10=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @11=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @12='13991172200' /* VARSTRING(600) meta=600 nullable=1 is_null=0 */
###   @13=1 /* TINYINT meta=0 nullable=1 is_null=0 */
###   @14='OPPO A95 全网通5G版' /* VARSTRING(240) meta=240 nullable=1 is_null=0 */
###   @15='炫黑 8GB+256GB 新装版' /* VARSTRING(2000) meta=2000 nullable=1 is_null=0 */
###   @16=NULL /* INT meta=0 nullable=1 is_null=1 */
###   @17=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @18=NULL /* VARSTRING(200) meta=200 nullable=1 is_null=1 */
###   @19='' /* VARSTRING(2000) meta=2000 nullable=1 is_null=0 */
--
###   @13='2023-07-19 09:50:27' /* DATETIME(0) meta=0 nullable=1 is_null=0 */
###   @14=NULL /* TIMESTAMP(0) meta=0 nullable=1 is_null=1 */
###   @15=0 /* TINYINT meta=0 nullable=0 is_null=0 */
###   @16=NULL /* DECIMAL(10,2) meta=2562 nullable=1 is_null=1 */
###   @17=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @18=NULL /* TINYINT meta=0 nullable=1 is_null=1 */
###   @19=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @20=NULL /* VARSTRING(200) meta=200 nullable=1 is_null=1 */
###   @21=NULL /* DATETIME(0) meta=0 nullable=1 is_null=1 */
###   @22=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @23=NULL /* DECIMAL(10,2) meta=2562 nullable=1 is_null=1 */
###   @24=NULL /* DECIMAL(10,2) meta=2562 nullable=1 is_null=1 */
###   @25=NULL /* DECIMAL(10,2) meta=2562 nullable=1 is_null=1 */
###   @26=NULL /* DECIMAL(10,2) meta=2562 nullable=1 is_null=1 */
###   @27=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @28=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @29=NULL /* DATETIME(0) meta=0 nullable=1 is_null=1 */
###   @30=NULL /* VARSTRING(1020) meta=1020 nullable=1 is_null=1 */
###   @31=0 /* TINYINT meta=0 nullable=0 is_null=0 */
# at 580667736
#230719  9:50:27 server id 32  end_log_pos 580667917 	Table_map: `small_oa_107`.`t_after_services` mapped to number 6956330
# has_generated_invisible_primary_key=0
# at 580667917
#230719  9:50:27 server id 32  end_log_pos 580668429 	Update_rows: table id 6956330 flags: STMT_END_F

And then, on 2023-07-20T10:10:28.107139+08:00 6239 the mysql group replication cluster stop, we can not start the secondary nodes, so we do a full clone via the 
mysql_clone plugin to recovery the mysql mgr cluster, it goes to normal as usually. but we found some data lost , by following binlog

#230720 10:23:03 server id 32  end_log_pos 902944493 	GTID	last_committed=322985	sequence_number=322986	rbr_only=yes	original_committed_timestamp=1689819783915613	immediate_commit_timestamp=1689819783916373	transaction_length=1615
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
# original_commit_timestamp=1689819783915613 (2023-07-20 10:23:03.915613 CST)
# immediate_commit_timestamp=1689819783916373 (2023-07-20 10:23:03.916373 CST)
/*!80001 SET @@session.original_commit_timestamp=1689819783915613*//*!*/;
/*!80014 SET @@session.original_server_version=80030*//*!*/;
/*!80014 SET @@session.immediate_server_version=80030*//*!*/;
SET @@SESSION.GTID_NEXT= 'cb1e163d-94e7-47e2-880f-82a481908bcf:67499714'/*!*/;
# at 902944493
#230720 10:23:03 server id 32  end_log_pos 902944572 	Query	thread_id=1104358	exec_time=0	error_code=0
SET TIMESTAMP=1689819783/*!*/;
BEGIN
/*!*/;
# at 902944572
#230720 10:23:03 server id 32  end_log_pos 902944753 	Table_map: `small_oa_107`.`t_after_services` mapped to number 10759830
# has_generated_invisible_primary_key=0
# at 902944753
#230720 10:23:03 server id 32  end_log_pos 902945002 	Write_rows: table id 10759830 flags: STMT_END_F
### INSERT INTO `small_oa_107`.`t_after_services`
### SET
###   @1=16820 /* LONGINT meta=0 nullable=0 is_null=0 */
###   @2=122078 /* LONGINT meta=0 nullable=0 is_null=0 */
###   @3='178683868xx' /* VARSTRING(44) meta=44 nullable=1 is_null=0 */
###   @4='178683868xx' /* VARSTRING(200) meta=200 nullable=1 is_null=0 */
###   @5=0 /* TINYINT meta=0 nullable=0 is_null=0 */
###   @6=NULL /* TIMESTAMP(0) meta=0 nullable=1 is_null=1 */
###   @7=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @8=9 /* LONGINT meta=0 nullable=0 is_null=0 */
###   @9=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @10=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @11=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @12='66698968' /* VARSTRING(600) meta=600 nullable=1 is_null=0 */
###   @13=1 /* TINYINT meta=0 nullable=1 is_null=0 */
###   @14='vivo Y73s 全网通5G版' /* VARSTRING(240) meta=240 nullable=1 is_null=0 */
###   @15='黑镜 8GB+128GB' /* VARSTRING(2000) meta=2000 nullable=1 is_null=0 */
###   @16=NULL /* INT meta=0 nullable=1 is_null=1 */
###   @17=NULL /* LONGINT meta=0 nullable=1 is_null=1 */
###   @18=NULL /* VARSTRING(200) meta=200 nullable=1 is_null=1 */
###   @19='' /* VARSTRING(2000) meta=2000 nullable=1 is_null=0 */

The new record with totally different content, inserted by the an used primary key 16820, without any delete record operation and ALTER TABLE AUTO_INCREMENT operation, we lost the 16820-16847 records forever.

How to repeat:
none

Suggested fix:
none

Here is the primary node log when mgr failure occered:

2023-07-20T09:47:44.585687+08:00 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: saas1:4306'
2023-07-20T09:47:44.592995+08:00 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to saas2:4306 on view 16824435424298041:7.'
2023-07-20T10:10:33.506079+08:00 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to saas2:4306, saas1:4306 on view 168244354242980
41:8.'
2023-07-20T10:10:34.429597+08:00 0 [Warning] [MY-011499] [Repl] Plugin group_replication reported: 'Members removed from the group: saas1:4306'
2023-07-20T10:10:34.429796+08:00 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to saas2:4306 on view 16824435424298041:9.'
2023-07-20T10:30:12.648155+08:00 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to saas2:4306, saas1:4306 on view 168244354242980
41:10.'
2023-07-20T10:30:23.873895+08:00 0 [System] [MY-011492] [Repl] Plugin group_replication reported: 'The member with address saas1:4306 was declared online within the replicat
ion group.'

Here is the secondary node's log when mgr cluster failure:

2023-07-20T10:10:28.107139+08:00 6239 [System] [MY-011651] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been stopped.'
2023-07-20T10:10:31.874754+08:00 6239 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2023-07-20T10:10:31.993837+08:00 6239 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv4 localhost address to the allowlist. It
 is mandatory that it is added.'
2023-07-20T10:10:31.993889+08:00 6239 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv6 localhost address to the allowlist. It
 is mandatory that it is added.'
2023-07-20T10:10:32.042327+08:00 6243 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', m
aster_port= 0, master_log_file='', master_log_pos= 3793174617, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_
bind=''.
2023-07-20T10:10:32.114867+08:00 6246 [ERROR] [MY-010584] [Repl] Slave SQL for channel 'group_replication_applier': Worker 1 failed executing transaction 'cb1e163d-94e7-47e2
-880f-82a481908bcf:67485087'; Could not execute Write_rows event on table small_oa_107.t_add_order_staff_model; Duplicate entry '11512' for key 't_add_order_staff_model.PRIM
ARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY, Error_code: MY-001062
2023-07-20T10:10:32.115034+08:00 6245 [Warning] [MY-010584] [Repl] Slave SQL for channel 'group_replication_applier': ... The slave coordinator and worker threads are stoppe
d, possibly leaving data in inconsistent state. A restart should restore consistency automatically, although using non-transactional storage for data or info tables or DDL q
ueries could lead to problems. In such cases you have to examine your data (see documentation for details). Error_code: MY-001756
2023-07-20T10:10:32.115099+08:00 6245 [ERROR] [MY-011451] [Repl] Plugin group_replication reported: 'The applier thread execution was aborted. Unable to process more transac
tions, this member will now leave the group.'
2023-07-20T10:10:32.115206+08:00 6243 [ERROR] [MY-011452] [Repl] Plugin group_replication reported: 'Fatal error during execution on the Applier process of Group Replication
. The server will now leave the group.'
2023-07-20T10:10:32.115358+08:00 6243 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member is already leaving or joining a group.'
2023-07-20T10:10:32.115451+08:00 6243 [ERROR] [MY-011644] [Repl] Plugin group_replication reported: 'Unable to confirm whether the server has left the group or not. Check pe
rformance_schema.replication_group_members to check group membership information.'
2023-07-20T10:10:32.115481+08:00 6243 [ERROR] [MY-011712] [Repl] Plugin group_replication reported: 'The server was automatically set into read only mode after an error was 
detected.'
2023-07-20T10:10:33.508382+08:00 0 [ERROR] [MY-011502] [Repl] Plugin group_replication reported: 'There was a previous plugin error while the member joined the group. The me
mber will now exit the group.'
2023-07-20T10:10:33.508449+08:00 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to saas1:4306 on view 16824435424298041:8.'
2023-07-20T10:10:33.508492+08:00 0 [ERROR] [MY-011486] [Repl] Plugin group_replication reported: 'Message received while the plugin is not ready, message discarded.'
2023-07-20T10:10:36.667361+08:00 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.'
2023-07-20T10:12:39.029095+08:00 6257 [Warning] [MY-013460] [InnoDB] Clone removing all user data for provisioning: Started
2023-07-20T10:13:05.558622+08:00 6258 [Warning] [MY-012111] [InnoDB] Trying to access missing tablespace 16881
2023-07-20T10:14:30.987409+08:00 6259 [Warning] [MY-012111] [InnoDB] Trying to access missing tablespace 48086
2023-07-20T10:14:47.516814+08:00 6259 [Warning] [MY-012111] [InnoDB] Trying to access missing tablespace 21099
2023-07-20T10:15:13.496883+08:00 6260 [Warning] [MY-012111] [InnoDB] Trying to access missing tablespace 53457
2023-07-20T10:15:47.287587+08:00 6266 [Warning] [MY-012111] [InnoDB] Trying to access missing tablespace 39902
2023-07-20T10:19:43.680419+08:00 6257 [Warning] [MY-013460] [InnoDB] Clone removing all user data for provisioning: Finished
2023-07-20T10:21:45.315163+08:00 6257 [ERROR] [MY-013462] [Server] Clone shutting down server as RESTART failed. Please start server to complete clone operation.
2023-07-20T10:21:47.374408+08:00 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 6257  user: 'clone_user'.
2023-07-20T10:21:47.374601+08:00 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 39  user: 'root'.
2023-07-20T10:21:50.501152+08:00 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30)  MySQL Community Server - GPL.
2023-07-20T10:21:57.302375+08:00 0 [Warning] [MY-011068] [Server] The syntax '--ssl=off' is deprecated and will be removed in a future release. Please use --tls-version=inva
lid instead.
2023-07-20T10:21:57.302437+08:00 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release.
2023-07-20T10:21:57.302446+08:00 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-repository' is deprecated and will be removed in a future release.
2023-07-20T10:21:57.302462+08:00 0 [Warning] [MY-011068] [Server] The syntax 'log_slave_updates' is deprecated and will be removed in a future release. Please use log_replic
a_updates instead.
2023-07-20T10:21:57.302478+08:00 0 [Warning] [MY-011069] [Server] The syntax '--transaction-write-set-extraction' is deprecated and will be removed in a future release.
2023-07-20T10:21:57.311553+08:00 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentic
ation_policy instead.
2023-07-20T10:21:57.311606+08:00 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 614273
2023-07-20T10:21:57.444558+08:00 0 [Warning] [MY-013907] [InnoDB] Deprecated configuration parameters innodb_log_file_size and/or innodb_log_files_in_group have been used to
 compute innodb_redo_log_capacity=2147483648. Please use innodb_redo_log_capacity instead.
2023-07-20T10:21:57.464122+08:00 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-07-20T10:22:02.945264+08:00 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-07-20T10:22:28.222612+08:00 0 [Warning] [MY-011302] [Server] Plugin mysqlx reported: 'Failed at SSL configuration: "SSL context is not usable without certificate and pr
ivate key"'
2023-07-20T10:22:28.222874+08:00 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30'  socket: '/var/run/mysqld/mysqld4306.sock'  port:
 4306  MySQL Community Server - GPL.
2023-07-20T10:22:28.222875+08:00 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2023-07-20T10:30:09.263466+08:00 9 [System] [MY-013587] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' is starting.'
2023-07-20T10:30:09.264679+08:00 10 [System] [MY-011565] [Repl] Plugin group_replication reported: 'Setting super_read_only=ON.'
2023-07-20T10:30:09.270779+08:00 9 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv4 localhost address to the allowlist. It is
 mandatory that it is added.'
2023-07-20T10:30:09.270827+08:00 9 [Warning] [MY-011735] [Repl] Plugin group_replication reported:

So we guess the problem is the mysql_clone plugin do a wrong gtid position adjustment to cause the data loss, or any other ideas/tips?

Hi Mr. Li,

Can you send us a fully repeatable test case. That means all the queries that lead to the loss of data and the complete description of the configuration.

We need a fully repeatable test case.

Here is the additional secondary node's log:

2023-07-20T10:30:09.270827+08:00 9 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Automatically adding IPv6 localhost address to the allowlist. It is
 mandatory that it is added.'
2023-07-20T10:30:09.277098+08:00 12 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', mas
ter_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2023-07-20T10:30:12.666793+08:00 9 [System] [MY-011511] [Repl] Plugin group_replication reported: 'This server is working as secondary member with primary member address saa
s2:4306.'
2023-07-20T10:30:13.667569+08:00 0 [System] [MY-013471] [Repl] Plugin group_replication reported: 'Distributed recovery will transfer data using: Incremental recovery from a
 group donor'
2023-07-20T10:30:13.706600+08:00 0 [System] [MY-011503] [Repl] Plugin group_replication reported: 'Group membership changed to saas2:4306, saas1:4306 on view 168244354242980
41:10.'
2023-07-20T10:30:13.711715+08:00 29 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='<NULL>', ma
ster_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='saas2', master_port= 4306, master_log_file='', master_log_pos= 4, master_bind=''.
2023-07-20T10:30:13.720162+08:00 30 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the master info repository is not secure and is therefore
 not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2023-07-20T10:30:13.721243+08:00 30 [System] [MY-010562] [Repl] Slave I/O thread for channel 'group_replication_recovery': connected to master 'mgruser@saas2:4306',replicati
on started in log 'FIRST' at position 4
2023-07-20T10:30:23.782702+08:00 29 [System] [MY-010597] [Repl] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='saas2', mas
ter_port= 4306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2023-07-20T10:30:23.851705+08:00 0 [System] [MY-011490] [Repl] Plugin group_replication reported: 'This server was declared online within the replication group.'

Hi Mr. LI,

We can't repeat your problem.

Hence, provide us with all the info that we have asked for.

Please, re-read our comment.

Here is the configration:

[root@saas2 ~]# cat /etc/my4306.cnf 
[client]  
default-character-set=utf8mb4  
  
[mysql]  
default-character-set = utf8mb4 

[mysqld]
#basedir=/data/mysql/data/
datadir=/data/mysql/data/4306
port=4306
user=mysql
pid-file	= /var/run/mysqld/mysqld4306.pid
socket		= /var/run/mysqld/mysqld4306.sock
default-time-zone = '+08:00'

lc_messages_dir=/usr/share/mysql-8.0/
lc_messages = en_US

character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4'
skip-character-set-client-handshake = true

max_connections = 5000
default-authentication-plugin=mysql_native_password
innodb_file_per_table=1
skip-ssl
explicit_defaults_for_timestamp

#performance
innodb_buffer_pool_size = 18800M
innodb_log_file_size=512M
innodb_log_files_in_group=4
max_heap_table_size = 256M
tmp_table_size = 256M
tmpdir=/data/mysql/data/temp4306
max_allowed_packet = 64M
slow_query_log=1
slow_query_log_file = /var/log/mysql4306-slow.log
log-error=/var/log/mysqld4306.log
long_query_time = 2
innodb_io_capacity=40000
innodb_io_capacity_max=50000
innodb_flush_method=O_DIRECT
innodb_flush_neighbors=0
group_replication_flow_control_applier_threshold=250000
group_replication_flow_control_certifier_threshold=250000

#binlog binlog config
server_id=32
gtid_mode=on
enforce_gtid_consistency=on
master_info_repository=table
relay_log_info_repository=table
binlog_checksum=none
log_slave_updates=on
log_bin=mysql-bin
binlog_format=row
relay-log=saas2-relay-bin

#group replication
transaction_write_set_extraction=XXHASH64
loose-group_replication_group_name="cb1e163d-94e7-47e2-880f-82a481908bcf"
loose-group_replication_start_on_boot=off
loose-group_replication_local_address= "192.168.254.32:7606"
loose-group_replication_group_seeds= "192.168.254.31:7606,192.168.254.32:7606"
loose-group_replication_bootstrap_group= off

and the insert sql be like:
INSERT INTO `small_oa_107`.`t_after_services`(`user_id`, `user_mobile`, `user_name`, `is_under_insurance`, `buy_time`, `buy_area_id`, `in_area_id`, `order_id`, `order_detail_id`, `mkc_id`, `imei`, `is_mobile`, `product_name`, `product_color`, `deal_method`, `deal_user_id`, `deal_user_name`, `comment`, `repair_in_source_id`, `after_services_price`, `payed_price`, `status`, `problem_comment`, `is_take`, `take_time`, `appearance`, `ppid`, `cur_area_id`, `cost_price`, `fixed_time`, `send_repair_time`, `in_user_id`, `in_user_name`, `take_user_id`, `take_user_name`, `create_time`, `update_time`, `is_delete`, `added_server_id`, `year_server_id`, `imei_check`, `appearance_description`, `now_area_id`, `voucher_record_id`, `log_comment`, `take_away_price`, `added_service_check`, `order_source`, `customer_type`, `examine_price`, `other_cost_price`, `discount_card_price`, `product_type`, `mountings`) VALUES ( 89491, '139911722xx', '先生', 0, NULL, NULL, 19, NULL, NULL, NULL, '139911722xx', 1, 'OPPO A95 全网通5G版', '炫黑 8GB+256GB 新装版', 1, 704, '路福鹏', '', NULL, 160.00, 160.00, 2, '内外屏幕碎', 1, '2023-07-19 12:12:44', 0, 126061, 19, 75.00, '2023-07-19 12:12:33', NULL, 704, '路福鹏', 704, '路福鹏', '2023-07-19 09:50:13', '2023-07-19 12:12:32', 0, NULL, NULL, 0, '磕碰掉漆', 19, NULL, '以修好了', 160.00, 0, 5, 0, 75.00, 0.00, NULL, NULL, '无配件');

INSERT INTO `small_oa_107`.`t_after_services`( `user_id`, `user_mobile`, `user_name`, `is_under_insurance`, `buy_time`, `buy_area_id`, `in_area_id`, `order_id`, `order_detail_id`, `mkc_id`, `imei`, `is_mobile`, `product_name`, `product_color`, `deal_method`, `deal_user_id`, `deal_user_name`, `comment`, `repair_in_source_id`, `after_services_price`, `payed_price`, `status`, `problem_comment`, `is_take`, `take_time`, `appearance`, `ppid`, `cur_area_id`, `cost_price`, `fixed_time`, `send_repair_time`, `in_user_id`, `in_user_name`, `take_user_id`, `take_user_name`, `create_time`, `update_time`, `is_delete`, `added_server_id`, `year_server_id`, `imei_check`, `appearance_description`, `now_area_id`, `voucher_record_id`, `log_comment`, `take_away_price`, `added_service_check`, `order_source`, `customer_type`, `examine_price`, `other_cost_price`, `discount_card_price`, `product_type`, `mountings`) VALUES ( 122078, '178683868xx', '178683868xx', 0, NULL, NULL, 9, NULL, NULL, NULL, '66698968', 1, 'vivo Y73s 全网通5G版', '黑镜 8GB+128GB', 1, 510, '任超斌', '', NULL, 230.00, 230.00, 2, '内屏显示异常', 1, '2023-07-20 11:46:46', 0, 95685, 9, 105.00, '2023-07-20 11:46:38', NULL, 510, '任超斌', 510, '任超斌', '2023-07-20 10:23:03', '2023-07-20 11:46:38', 0, NULL, NULL, 0, '划痕', 9, NULL, '更换总成屏', 230.00, 0, 5, 0, 105.00, 0.00, NULL, NULL, '无配件');

the table defination:
-- small_oa_107.t_after_services definition

CREATE TABLE `t_after_services` (
  `id` bigint NOT NULL AUTO_INCREMENT,
  `user_id` bigint NOT NULL COMMENT '客户编号',
  `user_mobile` varchar(11) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '客户电话',
  `user_name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '客户姓名',
  `is_under_insurance` tinyint(1) NOT NULL COMMENT '是否在保',
  `buy_time` timestamp NULL DEFAULT NULL COMMENT '购买时间',
  `buy_area_id` bigint DEFAULT NULL COMMENT '购买地区编码',
  `in_area_id` bigint NOT NULL COMMENT '接件地区编码',
  `order_id` bigint DEFAULT NULL COMMENT '关联订单单号',
  `order_detail_id` bigint DEFAULT NULL COMMENT '关联订单详细编号',
  `mkc_id` bigint DEFAULT NULL COMMENT '库存编号',
  `imei` varchar(150) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT '串号',
  `is_mobile` tinyint(1) DEFAULT NULL COMMENT '大小件标识',
  `product_name` varchar(60) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '商品名称',
  `product_color` varchar(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '商品规格',
  `deal_method` int DEFAULT NULL COMMENT '处理方式 1 自修 2 外送',
  `deal_user_id` bigint DEFAULT NULL COMMENT '处理人编号',
  `deal_user_name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '处理人名称',
  `comment` varchar(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '备注',
  `repair_in_source_id` bigint DEFAULT NULL COMMENT '外修渠道',
  `after_services_price` decimal(10,2) DEFAULT '0.00' COMMENT '维修费用',
  `payed_price` decimal(10,2) DEFAULT NULL COMMENT '已支付金额',
  `status` tinyint DEFAULT NULL COMMENT '处理状态 1 已受理 2 已修好 3 修不好',
  `problem_comment` varchar(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '故障描述',
  `is_take` tinyint(1) DEFAULT NULL COMMENT '是否取机 0 否 1 是',
  `take_time` timestamp NULL DEFAULT NULL COMMENT '取机时间',
  `appearance` tinyint DEFAULT NULL COMMENT '外观情况 1 完好 0 破损',
  `ppid` int DEFAULT NULL COMMENT '商品规格编号',
  `cur_area_id` bigint DEFAULT NULL COMMENT '当前所在地',
  `cost_price` decimal(10,2) DEFAULT NULL COMMENT '成本',
  `fixed_time` timestamp NULL DEFAULT NULL COMMENT '已修好日期',
  `send_repair_time` timestamp NULL DEFAULT NULL COMMENT '外送时间',
  `in_user_id` bigint DEFAULT NULL COMMENT '接机人工号',
  `in_user_name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '接机人姓名',
  `take_user_id` bigint DEFAULT NULL COMMENT '取机人编号',
  `take_user_name` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '取机人名称',
  `create_time` timestamp NULL DEFAULT CURRENT_TIMESTAMP COMMENT '接件时间',
  `update_time` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间',
  `is_delete` tinyint unsigned NOT NULL DEFAULT '0' COMMENT '删除标志（0：未删除，1：已删除）',
  `added_server_id` bigint DEFAULT NULL COMMENT '增值服务',
  `year_server_id` bigint DEFAULT NULL COMMENT '年包服务',
  `imei_check` tinyint DEFAULT NULL COMMENT '是否已验证串号 0  未验证 1、已验证',
  `appearance_description` varchar(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '外观描述',
  `now_area_id` bigint DEFAULT NULL COMMENT '当前地区编码',
  `voucher_record_id` bigint DEFAULT NULL COMMENT '记账主表id',
  `log_comment` varchar(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '维修方案',
  `take_away_price` decimal(10,2) NOT NULL DEFAULT '0.00' COMMENT '取机金额(原始金额)',
  `added_service_check` tinyint DEFAULT NULL COMMENT '是否出险验证（0：未验证，1：已验证）',
  `order_source` smallint NOT NULL DEFAULT '1' COMMENT '订单来源 OrderSellTypeEnum',
  `customer_type` smallint NOT NULL DEFAULT '0' COMMENT '客户类型 OrderSourceSearchEnum',
  `examine_price` decimal(10,2) DEFAULT NULL COMMENT '考核成本：维修单内维修配件考核成本成本之和',
  `other_cost_price` decimal(10,2) DEFAULT NULL COMMENT '其他成本（除配件外的成本总和）',
  `discount_card_price` decimal(10,2) DEFAULT NULL COMMENT '优惠券使用金额',
  `product_type` tinyint(1) DEFAULT NULL COMMENT '商品类型：0、销售机，1、陈列机，2、瑕疵机，3、良品',
  `mountings` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '附带配件',
  PRIMARY KEY (`id`),
  KEY `idx_imei` (`imei`) USING BTREE COMMENT '序列号索引',
  KEY `idx_take_time` (`take_time`) USING BTREE,
  KEY `idx_in_area_id` (`in_area_id`) USING BTREE,
  KEY `idx_order_detail_id` (`order_detail_id`) USING BTREE,
  KEY `idx_user_mobile` (`user_mobile`) USING BTREE,
  KEY `idx_ppid` (`ppid`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci COMMENT='售后';

Got it, but by the way, the " a fully repeatable test case." have an item menu, I can prepare for it as possible as I can.

Hi,

I spent quite some time ( more than a week ) trying to reproduce this with 8.0.32 and 8.0.34 using configuration and tables like yours with different scenarios and I cannot reproduce the problem.

> " a fully repeatable test case." have an item menu, I can prepare for it as possible as I can

If you can provide us with test case that will upon installation of a new cluster with 8.0.34 or 8.1.0 reproduce the problem that would be awesome as in that case we can determine where the bug is and fix it. With the info you provided there is just not enough data to reproduce the problem.

Thank you