Description:
Two members of the same group assign different GNOs to the same transaction: same origin, same original_commit_timestamp, two different GTIDs written to their binary logs.
Group Replication in multi-primary mode, in runs where a member leaves and rejoins during a rolling restart under write load. We tried without the rolling restart and could not reproduce it.
An error does not always follow. Both members keep certifying and committing against numbering that no longer matches, and the divergence might stay invisible: several runs ended with every member ONLINE, the disagreement present only in the binary logs.
Where a failure does occur it is on the group_replication_applier channel, MY-010584: Can't find record, Error_code: 1032, or Duplicate entry ... for key, Error_code: 1062. In that case, the member stops applying. The applier failure seems a consequence, not a symptom to rely on.
## The evidence
The same transaction, read out of two members' binary logs with `mysqlbinlog`. Same origin `server_id`, same `original_committed_timestamp`, two different GTIDs, for example:
```
--- node1 /mnt/nvme/GNOREPRO/docker_volumes/node1/mysql-bin.000004 at 26049102
#260806 23:09:42 server id 1 end_log_pos 26049184 GTID last_committed=20272 sequence_number=24233 rbr_only=yes original_committed_timestamp=1786057782101334 immediate_commit_timestamp=1786057782808637 transaction_length=924
SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:184399'/*!*/;
#260806 23:09:42 server id 1 end_log_pos 26049255 Query thread_id=30 exec_time=0 error_code=0
#260806 23:09:42 server id 1 end_log_pos 26049325 Table_map: `test`.`hot_update_state` mapped to number 88
#260806 23:09:42 server id 1 end_log_pos 26049999 Delete_rows: table id 88 flags: STMT_END_F
### DELETE FROM `test`.`hot_update_state`
### WHERE
### @1=267443
### @2=1
### @3='hotrule-572'
### @4='run_id=20260806T225432Z-feb832fa op=hotupdate router=2 seq=143534 rule_uid=hotrule-572 state=cafef00d...'
### @5='2026-08-06 23:09:25'
#260806 23:09:42 server id 1 end_log_pos 26050026 Xid = 48978
COMMIT/*!*/;
--- node6 /mnt/nvme/GNOREPRO/docker_volumes/node6/mysql-bin.000002 at 141641159
#260806 23:09:42 server id 1 end_log_pos 141641241 GTID last_committed=129924 sequence_number=133884 rbr_only=yes original_committed_timestamp=1786057782101334 immediate_commit_timestamp=1786057782743707 transaction_length=924
SET @@SESSION.GTID_NEXT= 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:184398'/*!*/;
#260806 23:09:42 server id 1 end_log_pos 141641312 Query thread_id=30 exec_time=0 error_code=0
#260806 23:09:42 server id 1 end_log_pos 141641382 Table_map: `test`.`hot_update_state` mapped to number 91
#260806 23:09:42 server id 1 end_log_pos 141642056 Delete_rows: table id 91 flags: STMT_END_F
### DELETE FROM `test`.`hot_update_state`
### WHERE
### @1=267443
### @2=1
### @3='hotrule-572'
### @4='run_id=20260806T225432Z-feb832fa op=hotupdate router=2 seq=143534 rule_uid=hotrule-572 state=cafef00d...'
### @5='2026-08-06 23:09:25'
#260806 23:09:42 server id 1 end_log_pos 141642083 Xid = 270982
COMMIT/*!*/;
```
`node6` calls it `:184398`, every other member calls it `:184399`. 5 transactions differ this way in that example run.
Three GNOs later, `node6` could not apply a write to the same table:
```
node6.log:2026-08-06T23:09:42.755260Z 31 [ERROR] [MY-010584] [Repl] Replica SQL for channel 'group_replication_applier': Worker 3 failed executing transaction 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:184402'; Could not execute Write_rows event on table test.hot_update_state; Duplicate entry '1-hotrule-253' for key 'hot_update_state.UQE_hot_update_state_org_id_rule_uid', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY, Error_code: MY-001062
```
How to repeat:
## How to reproduce
The attached tarball is self-contained and is the one that produced the runs above.
- file `gnorepro.tar.gz`
- sha256 `cdfc52cb6f413f0cbbab114025a57a564f93a7cade430be5a08a19221fc7bcc0`
### 1. A host
Any Linux host with Docker, roughly 20 GB of RAM and a filesystem with at least 60 GB free for the datadirs. Used here: AWS EC2 `i4i.2xlarge`, Ubuntu 22.04.
Docker stores its images under its own data root, usually the root filesystem and not the one `datadir_base` points at. The server image needs a few GB there.
### 2. Prerequisites
Docker, Go and screen installed, `go` on the `PATH`. On Ubuntu:
```bash
sudo apt-get update -qq
sudo apt-get install -y docker.io screen
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
curl -fsSLO https://go.dev/dl/go1.23.4.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.23.4.linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' | sudo tee /etc/profile.d/go.sh
```
Log out and back in so the docker group and the Go path take effect.
### 3. Unpack
```bash
tar xzf gnorepro.tar.gz
cd gnorepro
```
### 4. Point datadir to available disk
One key in `config.toml`, at the top of the file, has to match the host:
```toml
[lab]
datadir_base = "/mnt/nvme" # a filesystem with ~60GB free, not the root disk
```
Nothing outside `config.toml` needs editing, and every other knob the run uses is in that same file: write rates, block size, the emulated WAN, the churn timings, how many attempts and how long each one runs.
### 5. Run reproduction
```bash
./reproduce.sh --attempts 10
```
This executes up to ten attempts of twenty minutes. An attempt that does not reproduce rebuilds the group and starts over. The first one that does stops the loop.
Interestingly, I tested this on long runs and it happens as well, but I found it more reliable to run it in short bursts. Not sure how to explain that, but short bursts are the most repeatable way I found to hit it to ensure sharing a verifiable testcase.
Evidence lands in `<datadir_base>/GNOREPRO/repro-<timestamp>/`, and the run prints the two binary logs to open to validate the first GNO mismatch.
### What reproducer do
It builds a seven-member group in Docker, one mysqlrouter per member, with `netem` shaping the bridge into three datacentres.
Each attempt starts three screen sessions: the workload, the churn, and a detector that compares GNOs live so the attempt can stop early.
The workload writes to two cache tables at once. `test.multi_written` holds short-lived rows: `INSERT`, an `UPDATE` on 3 of every 10, a `DELETE` 280 ms later, over 100 keys shared by every router. `test.hot_update_state` holds 1,366 rows seeded at startup and rewritten all run by an upsert, in bursts that come back to the same row from a different router; 20% delete the row first.
The churn takes members out from 7 down to 1: `super_read_only=1`, `STOP GROUP_REPLICATION`, clean shutdown, 10 to 20 seconds offline, restart, rejoin, wait for seven `ONLINE`.
At the end every member's binary logs are decoded and every member is compared against every other, 21 pairs. That comparison decides the verdict.
## Considerations
- The lab emulates WAN latency between datacentres. That makes it far more likely, but it also reproduces without that latency, less often.
- The lab sets `group_replication_gtid_assignment_block_size` to 1000, which also raises the frequency. We see it at the default `1000000` as well, and at `1`. The block size appears related but does not explain it.
- The lab default is an orderly `STOP GROUP_REPLICATION` and a clean restart. A bare `SHUTDOWN` makes it happen more often, but it is not required.
- We have never reproduced it on load alone. Rolling restarts seem to be required.
- Reproduced on 8.4.8 and 8.4.11.
### Other profiles available for testing
Each one overrides a few `config.toml` keys for a single run, without editing the file:
```bash
./reproduce.sh --profile v8-4-11 # 8.4.11
./reproduce.sh --profile default-blocksize # block_size = 1000000
./reproduce.sh --profile block1 # block_size = 1
./reproduce.sh --profile no-netem # no emulated WAN
./reproduce.sh --profile no-drained # bare SHUTDOWN
./reproduce.sh --profile lowrate # a much lower write rate
```
