Description:
InnoDB may permanently skip an undo record when clustered-record purge fails during shutdown
I found a potential issue in the InnoDB purge path. If purging a delete-marked clustered record fails while the server is entering shutdown, the corresponding undo record may never be
retried after restart.
The relevant logic is in row_purge():
purged = row_purge_record(node, undo_rec, thr, updated_extern, thd);
if (purged || srv_shutdown_state.load() >= SRV_SHUTDOWN_PURGE) {
return;
}
Normally, when row_purge_record() returns false, the purge worker waits and retries the same record. However, if srv_shutdown_state has reached SRV_SHUTDOWN_PURGE, the function returns
immediately even though purged is still false.
At this point, the undo record has already been fetched and the global purge iterator has advanced. During shutdown, purge history can subsequently be truncated. After restart, there is no
eligible purge backlog that would cause the failed record to be processed again.
For a delete-mark operation, this can leave the delete-marked clustered record physically present permanently, even though it is no longer visible to SQL queries.
How to repeat:
I reproduced the issue on a MySQL 8.4.7 debug build using an MTR test with a debug injection point.
The test performs the following steps:
1. Stop the InnoDB purge threads.
2. Create an InnoDB table and insert one row.
3. Delete the row, producing a delete-mark purge record.
4. Resume purge.
5. Inject a failure immediately before removing the clustered record, simulating row_purge_del_mark() returning false.
6. Wait until the server reaches SRV_SHUTDOWN_PURGE.
7. Allow row_purge() to take the shutdown early-return path.
8. Perform a graceful restart.
9. Wait until purge has caught up.
10. Enable innodb_stats_include_delete_marked, run ANALYZE TABLE, and inspect persistent table statistics.
The observed results are:
purge_caught_up
1
logical_rows
0
physical_rows_including_delete_marked
1
This means:
- Purge has caught up and has no eligible backlog.
- The deleted row is not logically visible.
- The delete-marked clustered record is still physically present.
Without the injected purge failure, the physical row count after restart is 0.
### Expected Result
If purging the current undo record fails, the record should remain retryable across shutdown and restart. The clustered record should eventually be physically removed after the failure
condition is resolved.
### Actual Result
When the purge operation returns false during SRV_SHUTDOWN_PURGE, row_purge() returns without preserving or requeuing the failed record. The purge iterator has already advanced, so the
record is not retried after restart, leaving the delete-marked clustered record physically present.
### Impact
A transient purge failure, such as an out-of-file-space condition during shutdown, may cause delete-marked clustered records to be skipped permanently. This can lead to unreclaimed storage
and persistent table bloat.
Suggested fix:
The shutdown path should not treat a failed purge operation as completed. The current undo record should either remain in purge history or be explicitly preserved for retry after restart.
Description: InnoDB may permanently skip an undo record when clustered-record purge fails during shutdown I found a potential issue in the InnoDB purge path. If purging a delete-marked clustered record fails while the server is entering shutdown, the corresponding undo record may never be retried after restart. The relevant logic is in row_purge(): purged = row_purge_record(node, undo_rec, thr, updated_extern, thd); if (purged || srv_shutdown_state.load() >= SRV_SHUTDOWN_PURGE) { return; } Normally, when row_purge_record() returns false, the purge worker waits and retries the same record. However, if srv_shutdown_state has reached SRV_SHUTDOWN_PURGE, the function returns immediately even though purged is still false. At this point, the undo record has already been fetched and the global purge iterator has advanced. During shutdown, purge history can subsequently be truncated. After restart, there is no eligible purge backlog that would cause the failed record to be processed again. For a delete-mark operation, this can leave the delete-marked clustered record physically present permanently, even though it is no longer visible to SQL queries. How to repeat: I reproduced the issue on a MySQL 8.4.7 debug build using an MTR test with a debug injection point. The test performs the following steps: 1. Stop the InnoDB purge threads. 2. Create an InnoDB table and insert one row. 3. Delete the row, producing a delete-mark purge record. 4. Resume purge. 5. Inject a failure immediately before removing the clustered record, simulating row_purge_del_mark() returning false. 6. Wait until the server reaches SRV_SHUTDOWN_PURGE. 7. Allow row_purge() to take the shutdown early-return path. 8. Perform a graceful restart. 9. Wait until purge has caught up. 10. Enable innodb_stats_include_delete_marked, run ANALYZE TABLE, and inspect persistent table statistics. The observed results are: purge_caught_up 1 logical_rows 0 physical_rows_including_delete_marked 1 This means: - Purge has caught up and has no eligible backlog. - The deleted row is not logically visible. - The delete-marked clustered record is still physically present. Without the injected purge failure, the physical row count after restart is 0. ### Expected Result If purging the current undo record fails, the record should remain retryable across shutdown and restart. The clustered record should eventually be physically removed after the failure condition is resolved. ### Actual Result When the purge operation returns false during SRV_SHUTDOWN_PURGE, row_purge() returns without preserving or requeuing the failed record. The purge iterator has already advanced, so the record is not retried after restart, leaving the delete-marked clustered record physically present. ### Impact A transient purge failure, such as an out-of-file-space condition during shutdown, may cause delete-marked clustered records to be skipped permanently. This can lead to unreclaimed storage and persistent table bloat. Suggested fix: The shutdown path should not treat a failed purge operation as completed. The current undo record should either remain in purge history or be explicitly preserved for retry after restart.