Description:
Hi all：
recently,we use 8.0.27 MGR，when we start SECONDARY node，COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE is not 0,
and then switch SECONDARY to primary COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE still not 0.sometimes is 1, sometimes larger than 1.
mysql> select a.member_id,a.member_host,a.member_role,b.COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE from performance_schema.replication_group_members a, performance_schema.replication_group_member_stats b where a.MEMBER_ID=b.member_id;
+--------------------------------------+-------------+-------------+--------------------------------------------+
| member_id | member_host | member_role | COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE |
+--------------------------------------+-------------+-------------+--------------------------------------------+
| 6f450880-5e0c-11ec-9998-000c29e4ec4a | mgr10 | SECONDARY | 1 |
| 97a04402-5dae-11ec-9190-000c29a8c42a | mgr11 | PRIMARY | 0 |
+--------------------------------------+-------------+-------------+--------------------------------------------+
2 rows in set (0.00 sec)
mysql> show master status ;
+------------------+----------+--------------+------------------+----------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+----------------------------------------------+
| mysql-bin.000007 | 3132 | | | eb2ec3f5-5daf-11ec-95bc-000c29a8c42a:1-36273 |
+------------------+----------+--------------+------------------+----------------------------------------------+
1 row in set (0.00 sec)
mysql> select group_replication_set_as_primary('6f450880-5e0c-11ec-9998-000c29e4ec4a');
+--------------------------------------------------------------------------+
| group_replication_set_as_primary('6f450880-5e0c-11ec-9998-000c29e4ec4a') |
+--------------------------------------------------------------------------+
| Primary server switched to: 6f450880-5e0c-11ec-9998-000c29e4ec4a |
+--------------------------------------------------------------------------+
1 row in set (5.55 sec)
mysql> select a.member_id,a.member_host,a.member_role,b.COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE from performance_schema.replication_group_members a, performance_schema.replication_group_member_stats b where a.MEMBER_ID=b.member_id;
+--------------------------------------+-------------+-------------+--------------------------------------------+
| member_id | member_host | member_role | COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE |
+--------------------------------------+-------------+-------------+--------------------------------------------+
| 6f450880-5e0c-11ec-9998-000c29e4ec4a | mgr10 | PRIMARY | 1 |
| 97a04402-5dae-11ec-9190-000c29a8c42a | mgr11 | SECONDARY | 0 |
+--------------------------------------+-------------+-------------+--------------------------------------------+
We find commit cb094ae3612e65c7eda4702d3471e5cf00db9f1a
BUG#33067441: COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE IS NOT UPDATED DURING RECOVERY
modify and new add recovery's COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE in recovery stage.
1.Applier_handler::handle_event add recovery satge trx increment.
2.group_replication_trans_before_commit add recovery stage trx decrement.
We watch applier_module->get_pipeline_stats_member_collector()->m_transactions_waiting_apply memory address，
find value increment 1 when applier thread revoke Applier_handler::handle_event ,but not decrement 1.
Hardware watchpoint 13: *(0x7fff1c234580)
Old value = 0
New value = 1
But we find sometimes revoke Query_log_event::do_apply_event to commit view change transation，
trans_commit function cache_mngr is nullptr then return sucess，not invoke group_replication_trans_before_commit
to decrement this value，so value still 1. step like follow:
7936 DBUG_PRINT("info",
(gdb)
7938 binlog_cache_mngr *cache_mngr = thd_get_cache_mngr(thd);
(gdb)
7939 Transaction_ctx *trn_ctx = thd->get_transaction();
(gdb)
(gdb) p cache_mngr
$20 = (binlog_cache_mngr *) 0x0
7940 my_xid xid = trn_ctx->xid_state()->get_xid()->get_my_xid();
(gdb)
7941 bool stmt_stuff_logged = false;
(gdb)
7942 bool trx_stuff_logged = false;
(gdb)
7943 bool skip_commit = is_loggable_xa_prepare(thd);
(gdb)
7944 bool is_atomic_ddl = false;
(gdb)
7946 DBUG_PRINT("enter", ("thd: 0x%llx, all: %s, xid: %llu, cache_mngr: 0x%llx",
(gdb)
7954 if (cache_mngr == nullptr) {
(gdb)
7955 if (!skip_commit && ha_commit_low(thd, all)) return RESULT_ABORTED;
(gdb)
7956 return RESULT_SUCCESS;
(gdb)
7935 DBUG_TRACE;
(gdb)
8188 }
How to repeat:
sometimes start group_replication in SECONDARY node.
Description: Hi all： recently,we use 8.0.27 MGR，when we start SECONDARY node，COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE is not 0, and then switch SECONDARY to primary COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE still not 0.sometimes is 1, sometimes larger than 1. mysql> select a.member_id,a.member_host,a.member_role,b.COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE from performance_schema.replication_group_members a, performance_schema.replication_group_member_stats b where a.MEMBER_ID=b.member_id; +--------------------------------------+-------------+-------------+--------------------------------------------+ | member_id | member_host | member_role | COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE | +--------------------------------------+-------------+-------------+--------------------------------------------+ | 6f450880-5e0c-11ec-9998-000c29e4ec4a | mgr10 | SECONDARY | 1 | | 97a04402-5dae-11ec-9190-000c29a8c42a | mgr11 | PRIMARY | 0 | +--------------------------------------+-------------+-------------+--------------------------------------------+ 2 rows in set (0.00 sec) mysql> show master status ; +------------------+----------+--------------+------------------+----------------------------------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+------------------+----------------------------------------------+ | mysql-bin.000007 | 3132 | | | eb2ec3f5-5daf-11ec-95bc-000c29a8c42a:1-36273 | +------------------+----------+--------------+------------------+----------------------------------------------+ 1 row in set (0.00 sec) mysql> select group_replication_set_as_primary('6f450880-5e0c-11ec-9998-000c29e4ec4a'); +--------------------------------------------------------------------------+ | group_replication_set_as_primary('6f450880-5e0c-11ec-9998-000c29e4ec4a') | +--------------------------------------------------------------------------+ | Primary server switched to: 6f450880-5e0c-11ec-9998-000c29e4ec4a | +--------------------------------------------------------------------------+ 1 row in set (5.55 sec) mysql> select a.member_id,a.member_host,a.member_role,b.COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE from performance_schema.replication_group_members a, performance_schema.replication_group_member_stats b where a.MEMBER_ID=b.member_id; +--------------------------------------+-------------+-------------+--------------------------------------------+ | member_id | member_host | member_role | COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE | +--------------------------------------+-------------+-------------+--------------------------------------------+ | 6f450880-5e0c-11ec-9998-000c29e4ec4a | mgr10 | PRIMARY | 1 | | 97a04402-5dae-11ec-9190-000c29a8c42a | mgr11 | SECONDARY | 0 | +--------------------------------------+-------------+-------------+--------------------------------------------+ We find commit cb094ae3612e65c7eda4702d3471e5cf00db9f1a BUG#33067441: COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE IS NOT UPDATED DURING RECOVERY modify and new add recovery's COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE in recovery stage. 1.Applier_handler::handle_event add recovery satge trx increment. 2.group_replication_trans_before_commit add recovery stage trx decrement. We watch applier_module->get_pipeline_stats_member_collector()->m_transactions_waiting_apply memory address， find value increment 1 when applier thread revoke Applier_handler::handle_event ,but not decrement 1. Hardware watchpoint 13: *(0x7fff1c234580) Old value = 0 New value = 1 But we find sometimes revoke Query_log_event::do_apply_event to commit view change transation， trans_commit function cache_mngr is nullptr then return sucess，not invoke group_replication_trans_before_commit to decrement this value，so value still 1. step like follow: 7936 DBUG_PRINT("info", (gdb) 7938 binlog_cache_mngr *cache_mngr = thd_get_cache_mngr(thd); (gdb) 7939 Transaction_ctx *trn_ctx = thd->get_transaction(); (gdb) (gdb) p cache_mngr $20 = (binlog_cache_mngr *) 0x0 7940 my_xid xid = trn_ctx->xid_state()->get_xid()->get_my_xid(); (gdb) 7941 bool stmt_stuff_logged = false; (gdb) 7942 bool trx_stuff_logged = false; (gdb) 7943 bool skip_commit = is_loggable_xa_prepare(thd); (gdb) 7944 bool is_atomic_ddl = false; (gdb) 7946 DBUG_PRINT("enter", ("thd: 0x%llx, all: %s, xid: %llu, cache_mngr: 0x%llx", (gdb) 7954 if (cache_mngr == nullptr) { (gdb) 7955 if (!skip_commit && ha_commit_low(thd, all)) return RESULT_ABORTED; (gdb) 7956 return RESULT_SUCCESS; (gdb) 7935 DBUG_TRACE; (gdb) 8188 } How to repeat: sometimes start group_replication in SECONDARY node.