Description:
We first found that our replication was lagging significantly. But: SQL thread was still shown as Running: Yes:
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for source to send event
...
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
Last_Errno: 1755
Last_Error: Cannot execute the current event group in the parallel mode. Encountered event Anonymous_Gtid, relay-log name ./enswitch-db-00-42-relay-bin.007509, position 3612 which prevents execution of this event group in parallel mode. Reason: The source event is logically timestamped incorrectly..
...
Seconds_Behind_Master: 690603
...
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1755
Last_SQL_Error: Cannot execute the current event group in the parallel mode. Encountered event Anonymous_Gtid, relay-log name ./enswitch-db-00-42-relay-bin.007509, position 3612 which prevents execution of this event group in parallel mode. Reason: The source event is logically timestamped incorrectly..
...
Slave_SQL_Running_State: Waiting for workers to exit
...
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 250707 12:09:00
Error in MySQL Error Log:
2025-07-07T14:59:18.985754Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.42-0ubuntu0.24.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu).
2025-07-07T14:59:18.985825Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2025-07-07T14:59:18.987718Z 9 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2025-07-07T14:59:19.035439Z 9 [System] [MY-014001] [Repl] Replica receiver thread for channel '': connected to source 'repl@xxx:3306' with server_uuid=c3988eaa-a1d2-11ed-969c-525400977b42, server_id=100. Starting replication from file 'mysql-bin.289618', position '90125972'.
2025-07-07T14:59:21.454671Z 14 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'
2025-07-07T15:08:08.012430Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 50926934; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 50926934, Error_code: MY-001205
2025-07-07T15:08:59.020909Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 50926934; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 50926934, Error_code: MY-001205
2025-07-07T15:10:40.154090Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 51806805; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 51806805, Error_code: MY-001205
2025-07-07T15:11:31.165873Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 51806805; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 51806805, Error_code: MY-001205
2025-07-07T15:13:12.719169Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 56138033; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 56138033, Error_code: MY-001205
2025-07-07T16:09:00.533546Z 5 [ERROR] [MY-010411] [Repl] Transaction's sequence number is inconsistent with that of a preceding one: sequence_number (38002) <= previous sequence_number (39369)
2025-07-07T16:09:00.534502Z 5 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Coordinator thread of multi-threaded replica is being stopped in the middle of assigning a group of events; deferring to exit until the group completion ... , Error_code: MY-000000
2025-07-07T16:09:00.534518Z 5 [ERROR] [MY-010584] [Repl] Replica SQL for channel '': Cannot execute the current event group in the parallel mode. Encountered event Anonymous_Gtid, relay-log name ./xxx-db-00-42-relay-bin.007509, position 3612 which prevents execution of this event group in parallel mode. Reason: The source event is logically timestamped incorrectly.. Error_code: MY-001755
2025-07-07T16:09:00.534532Z 5 [Warning] [MY-010584] [Repl] Replica: Cannot execute the current event group in the parallel mode. Encountered event Anonymous_Gtid, relay-log name ./xxx-db-00-42-relay-bin.007509, position 3612 which prevents execution of this event group in parallel mode. Reason: The source event is logically timestamped incorrectly.. Error_code: MY-001755
The we tried to stop replication:
mysql> stop slave;
But stopping hangs forever:
mysql> show processlist;
+------+-----------------+----------------------------+------+-------------+--------+-----------------------------------------------------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+-----------------+----------------------------+------+-------------+--------+-----------------------------------------------------------------+------------------+
| 5 | system user | | NULL | Query | 65365 | Waiting for workers to exit | NULL |
| 6 | event_scheduler | localhost | NULL | Daemon | 69547 | Waiting on empty queue | NULL |
| 9 | system user | connecting host | NULL | Connect | 69547 | Waiting for source to send event | NULL |
| 10 | system user | | NULL | Query | 691044 | Waiting for preceding transaction to commit | NULL |
| 5484 | root | localhost | NULL | Killed | 6934 | Killing replica | stop slave |
| 5489 | repl | xxx-db-01-42.xxx.com:51936 | NULL | Binlog Dump | 6901 | Source has sent all binlog to replica; waiting for more updates | NULL |
| 5738 | root | localhost | NULL | Query | 0 | init | show processlist |
+------+-----------------+----------------------------+------+-------------+--------+-----------------------------------------------------------------+------------------+
I attache backtrace.
How to repeat:
No idea.
Suggested fix:
Make replication thread stop.
Description: We first found that our replication was lagging significantly. But: SQL thread was still shown as Running: Yes: mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event ... Slave_IO_Running: Yes Slave_SQL_Running: Yes ... Last_Errno: 1755 Last_Error: Cannot execute the current event group in the parallel mode. Encountered event Anonymous_Gtid, relay-log name ./enswitch-db-00-42-relay-bin.007509, position 3612 which prevents execution of this event group in parallel mode. Reason: The source event is logically timestamped incorrectly.. ... Seconds_Behind_Master: 690603 ... Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 1755 Last_SQL_Error: Cannot execute the current event group in the parallel mode. Encountered event Anonymous_Gtid, relay-log name ./enswitch-db-00-42-relay-bin.007509, position 3612 which prevents execution of this event group in parallel mode. Reason: The source event is logically timestamped incorrectly.. ... Slave_SQL_Running_State: Waiting for workers to exit ... Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: 250707 12:09:00 Error in MySQL Error Log: 2025-07-07T14:59:18.985754Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.42-0ubuntu0.24.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu). 2025-07-07T14:59:18.985825Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock 2025-07-07T14:59:18.987718Z 9 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information. 2025-07-07T14:59:19.035439Z 9 [System] [MY-014001] [Repl] Replica receiver thread for channel '': connected to source 'repl@xxx:3306' with server_uuid=c3988eaa-a1d2-11ed-969c-525400977b42, server_id=100. Starting replication from file 'mysql-bin.289618', position '90125972'. 2025-07-07T14:59:21.454671Z 14 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead' 2025-07-07T15:08:08.012430Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 50926934; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 50926934, Error_code: MY-001205 2025-07-07T15:08:59.020909Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 50926934; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 50926934, Error_code: MY-001205 2025-07-07T15:10:40.154090Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 51806805; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 51806805, Error_code: MY-001205 2025-07-07T15:11:31.165873Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 51806805; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 51806805, Error_code: MY-001205 2025-07-07T15:13:12.719169Z 11 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Worker 2 failed executing transaction 'ANONYMOUS' at source log mysql-bin.288371, end_log_pos 56138033; Could not execute Update_rows event on table xxx.subscriptions; Lock wait timeout exceeded; try restarting transaction, Error_code: 1205; handler error HA_ERR_LOCK_WAIT_TIMEOUT; the event's source log mysql-bin.288371, end_log_pos 56138033, Error_code: MY-001205 2025-07-07T16:09:00.533546Z 5 [ERROR] [MY-010411] [Repl] Transaction's sequence number is inconsistent with that of a preceding one: sequence_number (38002) <= previous sequence_number (39369) 2025-07-07T16:09:00.534502Z 5 [Warning] [MY-010584] [Repl] Replica SQL for channel '': Coordinator thread of multi-threaded replica is being stopped in the middle of assigning a group of events; deferring to exit until the group completion ... , Error_code: MY-000000 2025-07-07T16:09:00.534518Z 5 [ERROR] [MY-010584] [Repl] Replica SQL for channel '': Cannot execute the current event group in the parallel mode. Encountered event Anonymous_Gtid, relay-log name ./xxx-db-00-42-relay-bin.007509, position 3612 which prevents execution of this event group in parallel mode. Reason: The source event is logically timestamped incorrectly.. Error_code: MY-001755 2025-07-07T16:09:00.534532Z 5 [Warning] [MY-010584] [Repl] Replica: Cannot execute the current event group in the parallel mode. Encountered event Anonymous_Gtid, relay-log name ./xxx-db-00-42-relay-bin.007509, position 3612 which prevents execution of this event group in parallel mode. Reason: The source event is logically timestamped incorrectly.. Error_code: MY-001755 The we tried to stop replication: mysql> stop slave; But stopping hangs forever: mysql> show processlist; +------+-----------------+----------------------------+------+-------------+--------+-----------------------------------------------------------------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +------+-----------------+----------------------------+------+-------------+--------+-----------------------------------------------------------------+------------------+ | 5 | system user | | NULL | Query | 65365 | Waiting for workers to exit | NULL | | 6 | event_scheduler | localhost | NULL | Daemon | 69547 | Waiting on empty queue | NULL | | 9 | system user | connecting host | NULL | Connect | 69547 | Waiting for source to send event | NULL | | 10 | system user | | NULL | Query | 691044 | Waiting for preceding transaction to commit | NULL | | 5484 | root | localhost | NULL | Killed | 6934 | Killing replica | stop slave | | 5489 | repl | xxx-db-01-42.xxx.com:51936 | NULL | Binlog Dump | 6901 | Source has sent all binlog to replica; waiting for more updates | NULL | | 5738 | root | localhost | NULL | Query | 0 | init | show processlist | +------+-----------------+----------------------------+------+-------------+--------+-----------------------------------------------------------------+------------------+ I attache backtrace. How to repeat: No idea. Suggested fix: Make replication thread stop.