Description:
With the Change Stream Applier (APPLIER_VERSION=2) and
replica_preserve_commit_order=ON, a commit-order stall of 10 seconds or more
that ends with STOP REPLICA, or with a final apply failure of the stalled
transaction, causes two problems:
1. mysqld crashes. Debug builds abort on SIGSEGV or an assertion. Release
builds SIGSEGV in Job_applier::set_failure(), or keep running with a
corrupted table cache and SIGSEGV on the next FLUSH TABLES or at
shutdown.
2. replica_preserve_commit_order is violated. A transaction queued behind
the stalled one can commit before transactions that precede it in the
source's commit order. The replica's binary log records them in that
wrong order. The data becomes consistent again after the applier
restarts. Debug builds assert in Commit_order_manager::register_trx()
instead.
The stall can be caused by anything that blocks the next transaction to
commit for 10 s, for example a row lock held by a client on the replica.
Root cause
When the applier makes no progress for 10 s,
Applier_channel_monitor::check_applier_progress() requests an unblock, and
the scheduler dispatches the stalled transaction's commit task on a helper
thread (Dispatch_reason::unblocked, scheduler.cpp:207-216) while the owner
thread is still running its prepare phase. The helper registers the
transaction in the commit order queue (Job_applier::commit_register(),
job_applier.cpp:239-243), then waits for the owner to finish. Both threads
share one Job_applier and one session (THD). This breaks two assumptions:
1. Session ownership. Job_applier records only whether its session is
attached (m_is_attached), not which thread attached it, and detach()
ignores its thread_id. The helper's end_task (task_exec_job.cpp:79-97)
therefore detaches the session the owner is still using. When the
transaction fails (or STOP REPLICA kills the owner), the owner's
set_failure() marks the job done, the helper's poll at
job_applier.cpp:116-122 wakes, and the helper also runs set_failure()
on the owner's THD, with current_thd == nullptr, while the owner is
still in set_failure()/detach(). The helper either dereferences the
m_relay_context that the owner's detach() just reset (:182-185), or runs
cleanup_context() -> close_thread_tables() / innobase_commit() on a THD
it does not own.
2. Commit order queue. set_failure() does not leave the commit order queue
("to fail quickly, don't unregister from COM queue",
job_applier.cpp:514-515). That is only safe if a transaction that fails
in prepare never registered, which the unblocked dispatch breaks. The
failed transaction's entry stays at the queue front under the helper's
worker slot (a pool thread id). When the helper's thread runs the next
transaction, register_trx() pushes the same slot again (the asserts at
rpl_replica_commit_order_manager.cc:73 and commit_order_queue.cc:174,183
are debug-only), and wait_on_graph() (:106) sees "front == my slot" and
lets that transaction commit at once, ahead of the ones queued before
it.
The two are independent but happen together. Classic MTA (APPLIER_VERSION=1) and CSA with
replica_preserve_commit_order=OFF are not affected.
How to repeat:
1. Source and replica with GTID_MODE=ON, binlog_format=ROW. On the replica:
CHANGE REPLICATION SOURCE TO APPLIER_VERSION=2, APPLIER_WORKER_COUNT=4;
START REPLICA;
(replica_preserve_commit_order is ON by default.)
2. On the source:
CREATE TABLE test.t1 (id INT PRIMARY KEY);
Wait for the replica to catch up.
3. On the replica, in a separate session, keep this transaction open:
BEGIN; INSERT INTO test.t1 VALUES (1);
4. On the source, commit 10 transactions (GTIDs :2 to :11):
INSERT INTO test.t1 VALUES (1);
INSERT INTO test.t1 VALUES (2);
...
INSERT INTO test.t1 VALUES (10);
5. Wait 22 seconds (more than the 10 s monitor interval), then on the
replica:
STOP REPLICA;
SELECT @@GLOBAL.gtid_executed;
SELECT * FROM test.t1;
6. In the session from step 3: ROLLBACK. On the replica: START REPLICA,
wait for it to catch up, then FLUSH TABLES.
Result on trunk:
- Release build: at step 5, gtid_executed is <source_uuid>:1:5 (6 of 6
runs): :5 committed while :2-:4 did not. t1 holds only id=4 (the row of
:5), and the replica binlog has :5 right after :1. After the restart in
step 6 the data converges, then FLUSH TABLES crashes:
Table_cache::remove_table(TABLE*) sql/table_cache.h:490
Table_cache::free_all_unused_tables() sql/table_cache.cc:140
close_cached_tables(...) sql/sql_base.cc:1224
- Debug build: STOP REPLICA in step 5 crashes the replica:
Assertion failure: ha_innodb.cc:2823:EQ_CURRENT_THD(thd)
innobase_commit ha_innodb.cc:6010
Relay_log_info::cleanup_context() sql/rpl_rli.cc:1320
mysql::csa::Job_applier::set_failure() job_applier.cpp:519
end_task task_exec_job.cpp:84 <- :101
Suggested fix:
1. Session owner (fixes the crash). Job_applier records which pool thread
attached the session (std::atomic<Thread_id> m_session_owner), set in
attach() and cleared when that thread's task ends. detach() does nothing
for any other thread. A helper that does not own the session
(Job::is_bystander()) returns from end_task without set_done(),
set_success(), set_failure() or detach(); the owner reports, rolls back
and releases. The flags the helper polls become atomic.
bool Job_applier::detach(Thread_id thread_id) {
if (!m_is_attached || m_session_owner != thread_id) {
return false;
}
2. Leave the commit order queue on failure (fixes the reordering). The
helper waits until the owner's task has ended before its thread is
reused. New Commit_order_manager::finish_registered_after_failure(THD*),
called by set_failure() when the transaction is registered: if the
worker's slot is still REGISTERED, it calls set_rollback_status() and
removes the entry from wherever it is in the queue (as finish_one() does
for an overtaking transaction), passing the turn on if the entry was at
the front. It does not wait for the transaction's turn: after STOP
REPLICA the session is already killed, and an interrupted wait would
leave the slot REGISTERED and make ~Job_applier() wait forever for a
rollback that no one runs. Every transaction that gets its turn
afterwards rolls back with ER_REPLICA_WORKER_STOPPED_PREVIOUS_THD_ERROR,
and Relay_context::report_error() skips that error, as MTA's
report_error_to_coordinator() does, so Last_SQL_Errno keeps the original
error. Part 2 relies on part 1: only the owner may run set_failure().
Description: With the Change Stream Applier (APPLIER_VERSION=2) and replica_preserve_commit_order=ON, a commit-order stall of 10 seconds or more that ends with STOP REPLICA, or with a final apply failure of the stalled transaction, causes two problems: 1. mysqld crashes. Debug builds abort on SIGSEGV or an assertion. Release builds SIGSEGV in Job_applier::set_failure(), or keep running with a corrupted table cache and SIGSEGV on the next FLUSH TABLES or at shutdown. 2. replica_preserve_commit_order is violated. A transaction queued behind the stalled one can commit before transactions that precede it in the source's commit order. The replica's binary log records them in that wrong order. The data becomes consistent again after the applier restarts. Debug builds assert in Commit_order_manager::register_trx() instead. The stall can be caused by anything that blocks the next transaction to commit for 10 s, for example a row lock held by a client on the replica. Root cause When the applier makes no progress for 10 s, Applier_channel_monitor::check_applier_progress() requests an unblock, and the scheduler dispatches the stalled transaction's commit task on a helper thread (Dispatch_reason::unblocked, scheduler.cpp:207-216) while the owner thread is still running its prepare phase. The helper registers the transaction in the commit order queue (Job_applier::commit_register(), job_applier.cpp:239-243), then waits for the owner to finish. Both threads share one Job_applier and one session (THD). This breaks two assumptions: 1. Session ownership. Job_applier records only whether its session is attached (m_is_attached), not which thread attached it, and detach() ignores its thread_id. The helper's end_task (task_exec_job.cpp:79-97) therefore detaches the session the owner is still using. When the transaction fails (or STOP REPLICA kills the owner), the owner's set_failure() marks the job done, the helper's poll at job_applier.cpp:116-122 wakes, and the helper also runs set_failure() on the owner's THD, with current_thd == nullptr, while the owner is still in set_failure()/detach(). The helper either dereferences the m_relay_context that the owner's detach() just reset (:182-185), or runs cleanup_context() -> close_thread_tables() / innobase_commit() on a THD it does not own. 2. Commit order queue. set_failure() does not leave the commit order queue ("to fail quickly, don't unregister from COM queue", job_applier.cpp:514-515). That is only safe if a transaction that fails in prepare never registered, which the unblocked dispatch breaks. The failed transaction's entry stays at the queue front under the helper's worker slot (a pool thread id). When the helper's thread runs the next transaction, register_trx() pushes the same slot again (the asserts at rpl_replica_commit_order_manager.cc:73 and commit_order_queue.cc:174,183 are debug-only), and wait_on_graph() (:106) sees "front == my slot" and lets that transaction commit at once, ahead of the ones queued before it. The two are independent but happen together. Classic MTA (APPLIER_VERSION=1) and CSA with replica_preserve_commit_order=OFF are not affected. How to repeat: 1. Source and replica with GTID_MODE=ON, binlog_format=ROW. On the replica: CHANGE REPLICATION SOURCE TO APPLIER_VERSION=2, APPLIER_WORKER_COUNT=4; START REPLICA; (replica_preserve_commit_order is ON by default.) 2. On the source: CREATE TABLE test.t1 (id INT PRIMARY KEY); Wait for the replica to catch up. 3. On the replica, in a separate session, keep this transaction open: BEGIN; INSERT INTO test.t1 VALUES (1); 4. On the source, commit 10 transactions (GTIDs :2 to :11): INSERT INTO test.t1 VALUES (1); INSERT INTO test.t1 VALUES (2); ... INSERT INTO test.t1 VALUES (10); 5. Wait 22 seconds (more than the 10 s monitor interval), then on the replica: STOP REPLICA; SELECT @@GLOBAL.gtid_executed; SELECT * FROM test.t1; 6. In the session from step 3: ROLLBACK. On the replica: START REPLICA, wait for it to catch up, then FLUSH TABLES. Result on trunk: - Release build: at step 5, gtid_executed is <source_uuid>:1:5 (6 of 6 runs): :5 committed while :2-:4 did not. t1 holds only id=4 (the row of :5), and the replica binlog has :5 right after :1. After the restart in step 6 the data converges, then FLUSH TABLES crashes: Table_cache::remove_table(TABLE*) sql/table_cache.h:490 Table_cache::free_all_unused_tables() sql/table_cache.cc:140 close_cached_tables(...) sql/sql_base.cc:1224 - Debug build: STOP REPLICA in step 5 crashes the replica: Assertion failure: ha_innodb.cc:2823:EQ_CURRENT_THD(thd) innobase_commit ha_innodb.cc:6010 Relay_log_info::cleanup_context() sql/rpl_rli.cc:1320 mysql::csa::Job_applier::set_failure() job_applier.cpp:519 end_task task_exec_job.cpp:84 <- :101 Suggested fix: 1. Session owner (fixes the crash). Job_applier records which pool thread attached the session (std::atomic<Thread_id> m_session_owner), set in attach() and cleared when that thread's task ends. detach() does nothing for any other thread. A helper that does not own the session (Job::is_bystander()) returns from end_task without set_done(), set_success(), set_failure() or detach(); the owner reports, rolls back and releases. The flags the helper polls become atomic. bool Job_applier::detach(Thread_id thread_id) { if (!m_is_attached || m_session_owner != thread_id) { return false; } 2. Leave the commit order queue on failure (fixes the reordering). The helper waits until the owner's task has ended before its thread is reused. New Commit_order_manager::finish_registered_after_failure(THD*), called by set_failure() when the transaction is registered: if the worker's slot is still REGISTERED, it calls set_rollback_status() and removes the entry from wherever it is in the queue (as finish_one() does for an overtaking transaction), passing the turn on if the entry was at the front. It does not wait for the transaction's turn: after STOP REPLICA the session is already killed, and an interrupted wait would leave the slot REGISTERED and make ~Job_applier() wait forever for a rollback that no one runs. Every transaction that gets its turn afterwards rolls back with ER_REPLICA_WORKER_STOPPED_PREVIOUS_THD_ERROR, and Relay_context::report_error() skips that error, as MTA's report_error_to_coordinator() does, so Last_SQL_Errno keeps the original error. Part 2 relies on part 1: only the owner may run set_failure().