Description:
Hi,
Last_IO_Errno was changed from 1236 (in 5.7 and before) to 13114 (in 8.0 and later). However, Last_IO_Error still mentions "fatal error 1236", which is a confusing relic of the previous Last_IO_Errno. See How to repeat for details.
Note that How to repeat shows a manual / fake way to get "fatal error 1236". The blog post below explains how to get this error "in real life".
https://jfg-mysql.blogspot.com/2026/08/understanding-mysql-replication-fatal-error-1236.ht...
Last_IO_Errno and Last_IO_Error are fields of SHOW REPLICA STATUS, but this confusing relic is also found in at least two (2) other places as shown in How to repeat: 1) the P_S table replication_connection_status, and 2) the error log.
While we are at this, maybe it is an opportunity to improve the error message for "Replica has more GTIDs than the source has [...]". A minor improvement could be to remove the duplicate "the" (quoted below). A more important improvement could be to correct the reference to "rolled back transactions" (see quote below) because IMHO, there are no "actual" rollback, and the fact that transactions have disappeared from the source is a durability / persistence problem, not a "deliberate" rollback happening. There is also a problem with the recommended corrective actions, but I will make a dedicated follow-up report for this (I will add a comment once done). Finally, another minor improvement could be to make the error message fit the max error message length because it is currently truncated (see quote below). See Suggested fix for my take on the improved error message.
> [duplicate the] : [...] may indicate that the the last binary log file [...]
> [rolled back transactions] : [...] The source may have rolled back transactions that were already replicated to the replica. [...]
> [truncated] : [...] but are not included in GTID_EXECU
I realize that what I suggested is an incompatible change. Even if this affects 8.0, 8.4 and 9.7, some might claim it can only be fixed in the next IR release (following 26.7.0: 26.10.0 ?). However, because the follow-up report shows that the recommended recovery recommendations are wrong, it could be considered acceptable (and maybe even mandatory) to fix this in all releases.
Many thanks for looking into this,
Jean-François Gagné
How to repeat:
# Function to create and initialize a sandbox.
# (below is portable from MySQL 5.7 to 26.7.0, which is why we need sql_replica)
# (portability is also why we need cli_arg, more about this in Bug#118819)
# (the pv command is a trick to show execution time)
function create_and_init_sandbox() {
sql_replica=replica; grep -q -e mysql_5 <<< $v && sql_replica=slave
cli_arg=""; grep -q -e mysql_9.[4567] -e mysql_2 <<< $v && cli_arg=--commands
dbdeployer deploy replication $v -n 2 --gtid | pv -tN dbdepl. > /dev/null
cd ~/sandboxes/rsandbox_${v//./_}
./m <<< "CREATE DATABASE test_jfg"
./m <<< "CREATE TABLE test_jfg.t(id BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT)"
local i; for i in 1 2; do ./m <<< "INSERT INTO test_jfg.t(id) VALUE (NULL)"; done
}
# Function to manually inject a GTID on s1 and show the error after restarting replication.
# (DO SLEEP to let replication proceed)
function inject_gtid_on_replica_and_show_error() {
local gtid_executed="$(./m -N <<< "SELECT @@global.gtid_executed")"
local uuid="$(cut -d : -f 1 <<< "$gtid_executed")"
local ids="$( cut -d : -f 2 <<< "$gtid_executed")"
local last_id="$(cut -d - -f 2 <<< "$ids")"
local next_id=$(($last_id + 1))
./s1 <<< "SET GTID_NEXT = '$uuid:$next_id'; BEGIN; COMMIT"
local c; for c in m s1; do printf "%-2s: %s\n" $c $(./$c -N <<< "SELECT @@global.gtid_executed"); done
echo
./s1 $cli_arg <<< "STOP $sql_replica; START $sql_replica; DO SLEEP(1); SHOW $sql_replica STATUS\G" | grep -e IO_Running: -e Last_IO_Err..:
echo
./s1 -N <<< "SELECT LAST_ERROR_MESSAGE FROM performance_schema.replication_connection_status"
echo
local n=2; grep -q -e mysql_5 <<< $v && n=3
tail -n $n node1/data/msandbox.err
}
# Show the behavior for all versions.
# Last_IO_Errno was changed from 1236 to 13114 in 8.0, but Last_IO_Error still mentions "fatal error 1236".
# We can also find mentions of "fatal error 1236" in the P_S table replication_connection_status and the error log.
# (without the sleep 5 after create_and_init_sandbox, we might get errors because of a dbdeployer issue : https://github.com/ProxySQL/dbdeployer/issues/131)
for v in mysql_5.7.44 mysql_8.0.46 mysql_8.4.11 mysql_9.7.2 mysql_26.7.0; do
echo; echo $v; echo $v | tr "[:print:]" "-"
create_and_init_sandbox; sleep 5
echo; inject_gtid_on_replica_and_show_error
echo; ./stop_all | pv -tN stop_all > /dev/null
cd; rm -rf ~/sandboxes/rsandbox_${v//./_}
done | tail -n +2
mysql_5.7.44
------------
dbdepl.: 0:00:11
m : 00021145-1111-1111-1111-111111111111:1-20
s1: 00021145-1111-1111-1111-111111111111:1-21
Slave_IO_Running: No
Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been'
Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been'
2026-08-03T21:14:10.980650Z 9 [ERROR] Error reading packet from server for channel '': Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been (server_errno=1236)
2026-08-03T21:14:10.980667Z 9 [ERROR] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: 1236
2026-08-03T21:14:10.980670Z 9 [Note] Slave I/O thread exiting for channel '', read up to log 'mysql-bin.000001', position 4981
stop_all: 0:00:06
mysql_8.0.46
------------
dbdepl.: 0:00:11
m : 00023647-1111-1111-1111-111111111111:1-37
s1: 00023647-1111-1111-1111-111111111111:1-38
Replica_IO_Running: No
Last_IO_Errno: 13114
Last_IO_Error: Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU'
Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU'
2026-08-03T21:14:35.499540Z 19 [ERROR] [MY-010557] [Repl] Error reading packet from server for channel '': Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU (server_errno=1236)
2026-08-03T21:14:35.499557Z 19 [ERROR] [MY-013114] [Repl] Replica I/O for channel '': Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU', Error_code: MY-013114
stop_all: 0:00:06
mysql_8.4.11
------------
dbdepl.: 0:00:11
m : 00020512-1111-1111-1111-111111111111:1-37
s1: 00020512-1111-1111-1111-111111111111:1-38
Replica_IO_Running: No
Last_IO_Errno: 13114
Last_IO_Error: Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU'
Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU'
2026-08-03T21:14:59.114934Z 19 [ERROR] [MY-010557] [Repl] Error reading packet from server for channel '': Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU (server_errno=1236)
2026-08-03T21:14:59.114953Z 19 [ERROR] [MY-013114] [Repl] Replica I/O for channel '': Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU', Error_code: MY-013114
stop_all: 0:00:06
mysql_9.7.2
-----------
dbdepl.: 0:00:10
m : 00020903-1111-1111-1111-111111111111:1-37
s1: 00020903-1111-1111-1111-111111111111:1-38
Replica_IO_Running: No
Last_IO_Errno: 13114
Last_IO_Error: Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU'
Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU'
2026-08-03T21:15:22.391083Z 20 [ERROR] [MY-010557] [Repl] Error reading packet from server for channel '': Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU (server_errno=1236)
2026-08-03T21:15:22.391112Z 20 [ERROR] [MY-013114] [Repl] Replica I/O for channel '': Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU', Error_code: MY-013114
stop_all: 0:00:06
mysql_26.7.0
------------
dbdepl.: 0:00:10
m : 00037701-1111-1111-1111-111111111111:1-37
s1: 00037701-1111-1111-1111-111111111111:1-38
Replica_IO_Running: No
Last_IO_Errno: 13114
Last_IO_Error: Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU'
Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU'
2026-08-03T21:15:45.555125Z 20 [ERROR] [MY-010557] [Repl] Error reading packet from server for channel '': Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU (server_errno=1236)
2026-08-03T21:15:45.555151Z 20 [ERROR] [MY-013114] [Repl] Replica I/O for channel '': Got fatal error 1236 from source when reading data from binary log: 'Replica has more GTIDs than the source has, using the source's SERVER_UUID. This may indicate that the the last binary log file was truncated or lost, e.g., after a power failure when sync_binlog != 1. The source may have rolled back transactions that were already replicated to the replica. Replicate any transactions that source has rolled back from replica to source, and/or commit empty transactions on source to account for transactions that have been committed on source but are not included in GTID_EXECU', Error_code: MY-013114
stop_all: 0:00:05
Suggested fix:
Replace "Got fatal error 1236 from source when reading data from binary log" by "Got fatal error from source when reading data from binary log", or by the shorter "Fatal error from source" (shorter is better to avoid truncation).
Replace "indicate that the the last binary log" by "indicate that the last binary log", or by the shorter "indicate that the last binlog" (shorter is better to avoid truncation).
Consider removing everything except "Replica has more GTIDs than the source has, using the source's SERVER_UUID." and linking to a new page in the manual for explanations and corrective actions. If we do not want to have a link, we could simply add "This is an unusual situation where it might be unsafe to restart replication without a thorough investigation and corrective actions (a cause could be a source OS crash with sync_binlog != 1).".
