MySQL Bugs: #118043: mgr single leader mode tps slow down and 0 in the end when network is unstable

Bug #118043	mgr single leader mode tps slow down and 0 in the end when network is unstable
Submitted:	23 Apr 16:32	Modified:	11 Jun 12:17
Reporter:	pig zhu (OCA)	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server: Group Replication	Severity:	S2 (Serious)
Version:	8.0.36	OS:	CentOS
Assigned to:		CPU Architecture:	x86

Description:
In MGR's single leader mode, when network instability occurs (e.g., 50% packet loss rate), it may cause gradual performance degradation of the entire cluster until it drops to zero. This issue doesn't exist in non-single-leader modes.

Through code logging analysis, the problem sequence is identified as follows:

proposer_task acquires a site (site1) in the reserve_synode_number function and yields CPU during the while(too_far(*msgno)) loop.
executor_task gets scheduled, processes a view_msg message, generates a new site (site0), and elects a new leader. According to site0, the *msgno allocated in step 1 is ignored by executor_task.
proposer_task resumes execution and attempts to perform the propose action.
Since *msgno has been invalidated by executor_task, the acceptor_learner_task rejects responses.
proposer_task enters an infinite retry loop as it cannot complete the proposal.
Root cause: When using site for condition checks, the latest site value must be reacquired immediately before the check without any intervening yield operations. Network instability exacerbates this bug by triggering frequent site updates.

How to repeat:
1.build 3-nodes group, with autorejoin=0(for simplify this problem) and single_leader=ON
2.run sysbench write_only on primary
3.Repeatedly inject and restore network faults on the primary node。script is as follws:
for ((i=1; i<=20; i++))
do
    echo "========== Loop        $i  =========="

    sleep_time1=$(( RANDOM % 21 ))
    echo "First stage：sleep $sleep_time1 seconds"
    sleep $sleep_time1

    echo "inject network fault"
    sudo tc qdisc del dev bond0 root
    sudo tc qdisc show dev bond0
    sudo tc class show dev bond0
    sudo tc filter show dev bond0

    sudo tc qdisc add dev bond0 root handle 1: prio bands 4
    sudo tc qdisc add dev bond0 parent 1:4 handle 40: netem loss 50%
    sudo tc filter add dev bond0 parent 1: protocol ip prio 4 u32 match ip dst  *.*.*.*(ip of secondary 1)  flowid 1:4
    sudo tc filter add dev bond0 parent 1: protocol ip prio 4 u32 match ip dst  *.*.*.*(ip of secondary 2)  flowid 1:4

    sleep_time2=$(( RANDOM % 11 + 15 ))
    echo "Second stage：$sleep_time2 seconds"
    sleep $sleep_time2

    echo "can cel  network fault"

    sudo tc qdisc del dev bond0 root
    sudo tc qdisc show dev bond0
    sudo tc class show dev bond0
    sudo tc filter show dev bond0

    echo "-------------wait for group to recover---------------------"
    sleep 30
done 

In my test environment, the issue can be consistently reproduced within 10 iterations.

Suggested fix:
Check if msgno is ignored before retry_new in proposer_task.

@@ -2560,8 +2560,9 @@ static int proposer_task(task_arg arg) {
       while (!finished(ep->p)) { /* Try to get a value accepted */
         /* We will wake up periodically, and whenever a message arrives */
         TIMED_TASK_WAIT(&ep->p->rv, ep->delay = wakeup_delay(ep->delay));
+        ep->site = find_site_def_rw(ep->msgno);
         if (!synode_eq(ep->msgno, ep->p->synode) ||
-            ep->p->proposer.msg == nullptr) {
+          ep->p->proposer.msg == nullptr || ignore_message(ep->msgno, ep->site, "proposer_task")) {
           IFDBG(D_NONE, FN; STRLIT("detected stolen state machine, retry"););
           /* unlock_pax_machine(ep->p); */
           GOTO(retry_new); /* Need to break out of both loops,

Hi,

Can you reproduce this with 8.0.42? 8.0.36 is rather old release?

Thanks

Yes, this problem still exists on 8.0.42, and my suggested bug fix still works.

Yes, this problem still exists on 8.0.42, and my suggested bug fix still works.

Yes, this problem still exists on 8.0.42, and my suggested bug fix still works.

Thank you very much for your patch contribution, we appreciate it!

In order for us to continue the process of reviewing your contribution to MySQL, please send us a signed copy of the Oracle Contributor Agreement (OCA) as outlined in https://oca.opensource.oracle.com

Signing an OCA needs to be done only once and it's valid for all other Oracle governed Open Source projects as well.

Getting a signed/approved OCA on file will help us facilitate your contribution - this one, and others in the future.  

Please let me know, if you have any questions.

Thank you for your interest in MySQL.

done.

Thanks. Bug is verified.

This is duplicate of BUG#37764970, which is fixed in MySQL 8.0.43, 8.4.6, 9.4.0, 9.5.0 and upwards.