MySQL Bugs: #118515: Contribution by Tencent: trx_cleanup_at_db_startup did not clean up trx

Bug #118515	Contribution by Tencent: trx_cleanup_at_db_startup did not clean up trx_sys->shards
Submitted:	24 Jun 9:08	Modified:	30 Jun 10:48
Reporter:	yewei Xu (OCA)	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	8.4, 8.4.5	OS:	Any
Assigned to:		CPU Architecture:	Any
Tags:	Contribution

Description:
When MySQL crashes and restarts, it resurrects and inserts transactions that were committing before the crash into trx_sys->shards and trx_sys->rw_trx_list. In the crash recovery phase, if a transaction's state is already TRX_STATE_COMMITTED_IN_MEMORY, the transaction should be cleaned up. However, during this process, the transaction is only removed from trx_sys->rw_trx_list but not from trx_sys->shards. This can lead to stale trx_id entries remaining in trx_sys->shards, which may map to a transaction object (trx) that has already been reused. In a release build, this issue may lead to the following crash stack trace.

--------------------------------------------------------------------------------

#0 0x00007f7f991799b1 in pthread_kill () from /lib64/libpthread.so.0
#1 0x000000000149ea6b in handle_fatal_signal (sig=6) at /release/build/sql/signal_handler.cc:239
#2
#3 0x00007f7f97daa1f7 in raise () from /lib64/libc.so.6
#4 0x00007f7f97dab8e8 in abort () from /lib64/libc.so.6
#5 0x0000000000e50896 in my_server_abort () at /release/build/sql/signal_handler.cc:295
#6 0x000000000299a69a in my_abort () at /release/build/mysys/my_init.cc:258
#7 0x0000000002f7035c in ut_dbg_assertion_failed(char const*, char const*, unsigned long) () at /release/build/storage/innobase/ut/ut0dbg.cc:99
#8 0x0000000002d0ee2f in trx_reference (trx=0x7f7f71e1aba8) at /release/build/storage/innobase/include/trx0trx.ic:220
#9 trx_reference (trx=0x7f7f71e1aba8) at /release/build/storage/innobase/include/trx0trx.ic:218
#10 operator() (trx=0x7f7f71e1aba8, __closure=) at /release/build/storage/innobase/include/trx0sys.ic:229
#11 operator() (this=, this=, trx_by_id_with_min=...) at /release/build/storage/innobase/include/trx0sys.h:620
#12 latch_and_execute<trx_sys_t::latch_and_execute_with_active_trx<trx_rw_is_active(trx_id_t, bool="bool"> >::<lambda(trx_by_id_with_min&)> > (loc=..., loc=..., f=, this=0x7f7f692611e0) at /release/build/storage/innobase/include/ut0guarded.h:53
#13 latch_and_execute_with_active_trx<trx_rw_is_active(trx_id_t, bool="bool"> > (loc=..., f=, trx_id=22489, this=0x7f7f69253560) at /release/build/storage/innobase/include/trx0sys.h:618
#14 trx_rw_is_active (do_ref_count=true, trx_id=22489) at /release/build/storage/innobase/include/trx0sys.ic:225
#15 lock_rec_convert_impl_to_expl(buf_block_t const*, unsigned char const*, dict_index_t*, unsigned long const*, que_thr_t*) () at /release/build/storage/innobase/lock/lock0lock.cc:6784
#16 0x0000000002d0f23e in lock_clust_rec_read_check_and_lock(lock_duration_t, buf_block_t const*, unsigned char const*, dict_index_t*, unsigned long const*, select_mode, lock_mode, unsigned long, que_thr_t*) () at /release/build/storage/innobase/lock/lock0lock.cc:7018
#17 0x0000000002ec0807 in row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long) () at /release/build/storage/innobase/row/row0sel.cc:6164
#18 0x0000000002c69388 in ha_innobase::general_fetch (this=0x7f7ebbaf5030, buf=0x7f7ebbb04030 \377, , direction=1, match_mode=0) at /release/build/storage/innobase/handler/ha_innodb.cc:13464
#19 0x000000000168b838 in handler::ha_index_next (this=0x7f7ebbaf5030, buf=0x7f7ebbb04030 \377, ) at /release/build/sql/handler.cc:3616
#20 0x000000000168c32d in handler::read_range_next (this=0x7f7ebbaf5030) at /release/build/sql/handler.cc:7946
#21 0x0000000002c2409c in ha_innobase::read_range_next() () at /release/build/storage/innobase/handler/ha_innodb.cc:13749
#22 0x000000000168c6cd in handler::multi_range_read_next (this=0x7f7ebbaf5030, range_info=0x7f7f79536c18) at /release/build/sql/handler.cc:7039
#23 0x000000000168cc22 in handler::ha_multi_range_read_next (this=0x7f7ebbaf5030, range_info=range_info@entry=0x7f7f79536c18) at /release/build/sql/handler.cc:6976
#24 0x0000000001948f16 in IndexRangeScanIterator::Read (this=0x7f7ebba966c8) at /release/build/sql/range_optimizer/index_range_scan.cc:407
#25 0x00000000013d306f in Sql_cmd_update::update_single_table(THD*) () at /release/build/sql/sql_update.cc:892
#26 0x0000000001321aa7 in Sql_cmd_dml::execute(THD*) () at /release/build/sql/sql_select.cc:858
#27 0x000000000129e677 in mysql_execute_command(THD*, bool, bool) () at /release/build/sql/sql_parse.cc:5708
#28 0x00000000012a4294 in dispatch_sql_command (thd=thd@entry=0x7f7f76dc13c0, parser_state=parser_state@entry=0x7f7f795389d0) at /release/build/sql/sql_parse.cc:7907
#29 0x00000000012a7187 in dispatch_command(THD*, COM_DATA const*, enum_server_command) () at /release/build/sql/sql_parse.cc:3141
#30 0x00000000012aa6c4 in do_command (thd=0x7f7f76dc13c0) at /release/build/sql/sql_parse.cc:1903
#31 0x0000000001491fdc in handle_connection (arg=arg@entry=0x7f7ed1098280) at /release/build/sql/conn_handler/connection_handler_per_thread.cc:354
#32 0x00000000034a8333 in pfs_spawn_thread (arg=0x7f7ebd63ec20) at /release/build/storage/perfschema/pfs.cc:2942
#33 0x00007f7f99174e25 in start_thread () from /lib64/libpthread.so.0
#34 0x00007f7f97e6d35d in clone () from /lib64/libc.so.6

How to repeat:
Apply following  patch and run mtr test below, mysql will crash like:

2025-06-24T08:46:01.632386Z 8 [ERROR] [MY-013183] [InnoDB] Assertion failure: trx0trx.ic:59:state == TRX_STATE_NOT_STARTED || state == TRX_STATE_FORCED_ROLLBACK thread 140318128150272
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.4/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2025-06-24T08:46:01Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=6be0b44c6faf108d7a35470489e150636a8dcbda
Thread pointer: 0x7f9d94000da0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f9e5c2f09b0 thread_stack 0x100000
 #0 0x371103e print_fatal_signal at /data/integration/public/mysql-server-8.0/sql/signal_handler.cc:319
 #1 0x371138a _Z15my_server_abortv at /data/integration/public/mysql-server-8.0/sql/signal_handler.cc:447
 #2 0x4b9d766 _Z8my_abortv at /data/integration/public/mysql-server-8.0/mysys/my_init.cc:267
 #3 0x4fd11c0 _Z23ut_dbg_assertion_failedPKcS0_m at /data/integration/public/mysql-server-8.0/storage/innobase/ut/ut0dbg.cc:100
 #4 0x4d20dca trx_state_eq at /data/integration/public/mysql-server-8.0/storage/innobase/include/trx0trx.ic:59
 #5 0x4d3ffc4 _ZNK18Trx_by_id_with_min3getEm at /data/integration/public/mysql-server-8.0/storage/innobase/include/trx0sys.h:408
 #6 0x4d21308 operator() at /data/integration/public/mysql-server-8.0/storage/innobase/include/trx0sys.h:577
 #7 0x4d21369 latch_and_execute<trx_sys_t::latch_and_execute_with_active_trx<trx_rw_is_active(trx_id_t, bool)::<lambda(trx_t*)> >(trx_id_t, trx_rw_is_active(trx_id_t, bool)::<lambda(trx_t*)>&&, const ut::Location&)::<lambda(Trx_by_id_with_min&)> > at /data/integration/public/mysql-server-8.0/storage/innobase/include/ut0guarded.h:54
 #8 0x4d213f0 latch_and_execute_with_active_trx<trx_rw_is_active(trx_id_t, bool)::<lambda(trx_t*)> > at /data/integration/public/mysql-server-8.0/storage/innobase/include/trx0sys.h:575
 #9 0x4d214b7 trx_rw_is_active at /data/integration/public/mysql-server-8.0/storage/innobase/include/trx0sys.ic:218
 #10 0x4d3337f _Z29lock_rec_convert_impl_to_explPK11buf_block_tPKhP12dict_index_tPKm at /data/integration/public/mysql-server-8.0/storage/innobase/lock/lock0lock.cc:5325
 #11 0x4d34257 _Z34lock_clust_rec_read_check_and_lock15lock_duration_tPK11buf_block_tPKhP12dict_index_tPKm11select_mode9lock_modemP9que_thr_t at /data/integration/public/mysql-server-8.0/storage/innobase/lock/lock0lock.cc:5538

-------------------------------------patch---------------------------------------

diff --git a/storage/innobase/trx/trx0undo.cc b/storage/innobase/trx/trx0undo.cc
index 85da30b4ddf..49c1f706d2d 100644
--- a/storage/innobase/trx/trx0undo.cc
+++ b/storage/innobase/trx/trx0undo.cc
@@ -1813,8 +1813,10 @@ page_t *trx_undo_set_state_at_finish(
   seg_hdr = undo_page + TRX_UNDO_SEG_HDR;
   page_hdr = undo_page + TRX_UNDO_PAGE_HDR;
 
-  if (undo->size == 1 && mach_read_from_2(page_hdr + TRX_UNDO_PAGE_FREE) <
-                             TRX_UNDO_PAGE_REUSE_LIMIT) {
+  if (undo->size == 1 &&
+      mach_read_from_2(page_hdr + TRX_UNDO_PAGE_FREE) <
+          TRX_UNDO_PAGE_REUSE_LIMIT &&
+      DBUG_EVALUATE_IF("trx_undo_no_cache", false, true)) {
     state = TRX_UNDO_CACHED;
 
   } else if (undo->type == TRX_UNDO_INSERT) {

-------------------------------------mtr-----------------------------------------

--source include/have_log_bin.inc
--source include/have_debug.inc

create table t1 (id int primary key auto_increment);

insert into t1 values();

--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
SET SESSION debug='+d,ib_crash_during_trx_commit_in_mem';
SET SESSION debug='+d,trx_undo_no_cache';
# 2013 - CR_SERVER_LOST
--error 2013
insert into t1 values();
--source include/wait_until_disconnected.inc

--echo # Restart the master server
--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
--source include/wait_until_connected_again.inc

let SEARCH_FILE= $MYSQLTEST_VARDIR/log/mysqld.1.err;
let SEARCH_PATTERN= Cleaning up trx with id;
--source include/search_pattern.inc

update t1 set id = 3 where id = 2;

drop table t1;

Suggested fix:
diff --git a/storage/innobase/trx/trx0trx.cc b/storage/innobase/trx/trx0trx.cc
index 5481bd111ca..073c3304e9b 100644
--- a/storage/innobase/trx/trx0trx.cc
+++ b/storage/innobase/trx/trx0trx.cc
@@ -2303,6 +2303,17 @@ void trx_cleanup_at_db_startup(trx_t *trx) /*!< in: transaction */
 
   trx_sys_mutex_exit();
 
+  if (trx->id > 0) {
+    trx_sys->get_shard_by_trx_id(trx->id).active_rw_trxs.latch_and_execute(
+        [&](Trx_by_id_with_min &trx_by_id_with_min) {
+          ut_d(const size_t trx_shard_no = trx_get_shard_no(trx->id));
+          ut_ad(trx_get_shard_no(trx_by_id_with_min.min_id()) == trx_shard_no);
+          trx_by_id_with_min.erase(trx->id);
+          ut_ad(trx_get_shard_no(trx_by_id_with_min.min_id()) == trx_shard_no);
+        },
+        UT_LOCATION_HERE);
+  }
+
   /* Change the transaction state without mutex protection, now
   that it no longer is in the trx_list. Recovered transactions
   are never placed in the mysql_trx_list. */

Hello  Yewei Xu,

Thank you for the report and contribution.
Please ensure to re-send the patch via "Contribution" tab otherwise we would not be able to accept it. Thank you.

regards,
Umesh

-- 8.4.5 
-- Apply provided patch on top of 8.4.5 and build

scl enable gcc-toolset-12 bash
MYSQL_VERSION="Bug118515"
TARGET=/bv/ushastry/Work/Source/$MYSQL_VERSION

rm -rf /bv/ushastry/Work/Source/$MYSQL_VERSION
rm -rf bld/

mkdir bld && cd bld
rm -rf CMakeCache.txt

cmake .. -DBUILD_CONFIG=mysql_release -DWITH_DEBUG=1 -DWITH_UNIT_TESTS=0  -DCMAKE_INSTALL_PREFIX=$TARGET -G Ninja
ninja -j8
ninja install

-
[umshastr@bug118403:/bv/ushastry/Work/Source/Bug118515/mysql-test]$ ./mtr bug118515
Logging: ./mtr  bug118515
MySQL Version 8.4.5
Checking supported features
 - Binaries are debug compiled
Using 'all' suites
Collecting tests
Removing old var directory
Creating var directory '/bv/ushastry/Work/Source/Bug118515/mysql-test/var'
Installing system database
Using parallel: 1

==============================================================================
                  TEST NAME                       RESULT  TIME (ms) COMMENT
------------------------------------------------------------------------------
[ 50%] main.bug118515                            [ fail ]
        Test ended at 2025-06-30 12:47:32

CURRENT_TEST: main.bug118515
mysqltest: At line 24: Query 'update t1 set id = 3 where id = 2' failed.
ERROR 2013 (HY000): Lost connection to MySQL server during query
safe_process[86692]: Child process: 86693, exit: 1

.. excerpts from error log

2025-06-30T10:47:27.909386Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 13020, socket: /bv/ushastry/Work/Source/Bug118515/mysql-test/var/tmp/mysqlx.1.sock
2025-06-30T10:47:27.909671Z 0 [System] [MY-010931] [Server] /bv/ushastry/Work/Source/Bug118515/bin/mysqld: ready for connections. Version: '8.4.5-debug'  socket: '/bv/ushastry/Work/Source/Bug118515/mysql-test/var/tmp/mysqld.1.sock'  port: 13000  Source distribution.
2025-06-30T10:47:28.036450Z 8 [ERROR] [MY-013183] [InnoDB] Assertion failure: trx0trx.ic:59:state == TRX_STATE_NOT_STARTED || state == TRX_STATE_FORCED_ROLLBACK thread 140047948146240
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.4/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2025-06-30T10:47:28Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=5291e0e638adcda180cee2f2c629fd305ab7526c
Thread pointer: 0x7f5eb4001040
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f5f743359f0 thread_stack 0x100000
 #0 0x387a8ab print_fatal_signal at /bv/ushastry/Work/Source/mysql-8.4.5/sql/signal_handler.cc:319
 #1 0x387abf7 _Z15my_server_abortv at /bv/ushastry/Work/Source/mysql-8.4.5/sql/signal_handler.cc:447
 #2 0x4d63c28 _Z8my_abortv at /bv/ushastry/Work/Source/mysql-8.4.5/mysys/my_init.cc:267
 #3 0x51a6636 _Z23ut_dbg_assertion_failedPKcS0_m at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/ut/ut0dbg.cc:100
 #4 0x4eee88c trx_state_eq at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0trx.ic:59
 #5 0x4f0dc2e _ZNK18Trx_by_id_with_min3getEm at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0sys.h:408
 #6 0x4eeedca operator() at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0sys.h:577
 #7 0x4eeee2b latch_and_execute<trx_sys_t::latch_and_execute_with_active_trx<trx_rw_is_active(trx_id_t, bool)::<lambda(trx_t*)> >(trx_id_t, trx_rw_is_active(trx_id_t, bool)::<lambda(trx_t*)>&&, const ut::Location&)::<lambda(Trx_by_id_with_min&)> > at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/ut0guarded.h:54
 #8 0x4eeeeb2 latch_and_execute_with_active_trx<trx_rw_is_active(trx_id_t, bool)::<lambda(trx_t*)> > at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0sys.h:575
 #9 0x4eeef79 trx_rw_is_active at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0sys.ic:218
 #10 0x4f00f4a _Z29lock_rec_convert_impl_to_explPK11buf_block_tPKhP12dict_index_tPKm at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/lock/lock0lock.cc:5325
 #11 0x4f01e22 _Z34lock_clust_rec_read_check_and_lock15lock_duration_tPK11buf_block_tPKhP12dict_index_tPKm11select_mode9lock_modemP9que_thr_t at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/lock/lock0lock.cc:5538
 #12 0x50bd4f6 sel_set_rec_lock at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/row/row0sel.cc:1163
 #13 0x50c79ef _Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/row/row0sel.cc:5244
 #14 0x4df14d3 _ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/handler/ha_innodb.cc:10536
 #15 0x3a4580f _ZN7handler14index_read_mapEPhPKhm16ha_rkey_function at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.h:5621
 #16 0x3a3388b _ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:3287
 #17 0x3a3f254 _ZN7handler16read_range_firstEPK9key_rangeS2_bb at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:7405
 #18 0x4df2b9a _ZN11ha_innobase16read_range_firstEPK9key_rangeS2_bb at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/handler/ha_innodb.cc:11024
 #19 0x3a3d003 _ZN7handler21multi_range_read_nextEPPc at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:6553
 #20 0x3a3dfe9 _ZN10DsMrr_impl10dsmrr_nextEPPc at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:6904
 #21 0x4e0e3ff _ZN11ha_innobase21multi_range_read_nextEPPc at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/handler/ha_innodb.cc:23701
 #22 0x3a3cd35 _ZN7handler24ha_multi_range_read_nextEPPc at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:6491
 #23 0x3d4c03e _ZN22IndexRangeScanIterator4ReadEv at /bv/ushastry/Work/Source/mysql-8.4.5/sql/range_optimizer/index_range_scan.cc:374
 #24 0x377198c _ZN14Sql_cmd_update19update_single_tableEP3THD at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_update.cc:876
 #25 0x3774ae9 _ZN14Sql_cmd_update13execute_innerEP3THD at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_update.cc:1824
 #26 0x36b6daf _ZN11Sql_cmd_dml7executeEP3THD at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_select.cc:782
 #27 0x362c57f _Z21mysql_execute_commandP3THDb at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_parse.cc:3672
 #28 0x3631f4f _Z20dispatch_sql_commandP3THDP12Parser_state at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_parse.cc:5406
 #29 0x3627f12 _Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_parse.cc:2136
 #30 0x3625d5b _Z10do_commandP3THD at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_parse.cc:1465
 #31 0x3864f0f handle_connection at /bv/ushastry/Work/Source/mysql-8.4.5/sql/conn_handler/connection_handler_per_thread.cc:304
 #32 0x59e8c0d pfs_spawn_thread at /bv/ushastry/Work/Source/mysql-8.4.5/storage/perfschema/pfs.cc:3061
 #33 0x7f5f7f0bc1a9 <unknown>
 #34 0x7f5f7f14128f <unknown>
 #35 0xffffffffffffffff <unknown>

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f5eb400bae0): update t1 set id = 3 where id = 2
Connection ID (thread ID): 8
Status: NOT_KILLED

--bt

Thread 1 (Thread 0x7f5f74336640 (LWP 86746)):
#0  0x00007f5f7f0bdf1c in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000000004d6d068 in my_write_core (sig=6) at /bv/ushastry/Work/Source/mysql-8.4.5/mysys/stacktrace.cc:344
#2  0x000000000387ab7e in handle_fatal_signal (sig=6, info=0x7f5f7432f870, ucontext=0x7f5f7432f740) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/signal_handler.cc:409
#3  <signal handler called>
#4  0x00007f5f7f0bdf1c in __pthread_kill_implementation () from /lib64/libc.so.6
#5  0x00007f5f7f070b46 in raise () from /lib64/libc.so.6
#6  0x00007f5f7f05a833 in abort () from /lib64/libc.so.6
#7  0x000000000387ad0c in my_server_abort () at /bv/ushastry/Work/Source/mysql-8.4.5/sql/signal_handler.cc:465
#8  0x0000000004d63c29 in my_abort () at /bv/ushastry/Work/Source/mysql-8.4.5/mysys/my_init.cc:267
#9  0x00000000051a6637 in ut_dbg_assertion_failed (expr=0x74d5c50 "state == TRX_STATE_NOT_STARTED || state == TRX_STATE_FORCED_ROLLBACK", file=0x74d5bd8 "/bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0trx.ic", line=59) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/ut/ut0dbg.cc:100
#10 0x0000000004eee88d in trx_state_eq (trx=0x7f5f7746d428, state=TRX_STATE_COMMITTED_IN_MEMORY) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0trx.ic:59
#11 0x0000000004f0dc2f in Trx_by_id_with_min::get (this=0x7f5f703791f8, trx_id=1557) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0sys.h:408
#12 0x0000000004eeedcb in operator() (__closure=0x7f5f743308c0, trx_by_id_with_min=...) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0sys.h:577
#13 0x0000000004eeee2c in ut::Guarded<Trx_by_id_with_min, (latch_id_t)79>::latch_and_execute<trx_sys_t::latch_and_execute_with_active_trx<trx_rw_is_active(trx_id_t, bool)::<lambda(trx_t*)> >(trx_id_t, trx_rw_is_active(trx_id_t, bool)::<lambda(trx_t*)>&&, const ut::Location&)::<lambda(Trx_by_id_with_min&)> >(struct {...} &&, const ut::Location &) (this=0x7f5f70379140, f=..., loc=...) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/ut0guarded.h:54
#14 0x0000000004eeeeb3 in trx_sys_t::latch_and_execute_with_active_trx<trx_rw_is_active(trx_id_t, bool)::<lambda(trx_t*)> >(trx_id_t, struct {...} &&, const ut::Location &) (this=0x7f5f703773b0, trx_id=1557, f=..., loc=...) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0sys.h:575
#15 0x0000000004eeef7a in trx_rw_is_active (trx_id=1557, do_ref_count=true) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/include/trx0sys.ic:218
#16 0x0000000004f00f4b in lock_rec_convert_impl_to_expl (block=0x7f5f750f4948, rec=0x7f5f75eb0093 "", index=0x7f5eb40c25b8, offsets=0x7f5f74331100) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/lock/lock0lock.cc:5325
#17 0x0000000004f01e23 in lock_clust_rec_read_check_and_lock (duration=lock_duration_t::REGULAR, block=0x7f5f750f4948, rec=0x7f5f75eb0093 "", index=0x7f5eb40c25b8, offsets=0x7f5f74331100, sel_mode=SELECT_ORDINARY, mode=LOCK_X, gap_mode=1024, thr=0x7f5eb40c9848) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/lock/lock0lock.cc:5538
#18 0x00000000050bd4f7 in sel_set_rec_lock (pcur=0x7f5eb40c9160, rec=0x7f5f75eb0093 "", index=0x7f5eb40c25b8, offsets=0x7f5f74331100, sel_mode=SELECT_ORDINARY, mode=3, type=1024, thr=0x7f5eb40c9848, mtr=0x7f5f74331430) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/row/row0sel.cc:1163
#19 0x00000000050c79f0 in row_search_mvcc (buf=0x7f5eb409d550 "\377", mode=PAGE_CUR_GE, prebuilt=0x7f5eb40c8ef8, match_mode=1, direction=0) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/row/row0sel.cc:5244
#20 0x0000000004df14d4 in ha_innobase::index_read (this=0x7f5eb409d240, buf=0x7f5eb409d550 "\377", key_ptr=0x7f5eb40a7d08 "\002", key_len=4, find_flag=HA_READ_KEY_EXACT) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/handler/ha_innodb.cc:10536
#21 0x0000000003a45810 in handler::index_read_map (this=0x7f5eb409d240, buf=0x7f5eb409d550 "\377", key=0x7f5eb40a7d08 "\002", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.h:5621
#22 0x0000000003a3388c in handler::ha_index_read_map (this=0x7f5eb409d240, buf=0x7f5eb409d550 "\377", key=0x7f5eb40a7d08 "\002", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:3287
#23 0x0000000003a3f255 in handler::read_range_first (this=0x7f5eb409d240, start_key=0x7f5eb409d320, end_key=0x7f5eb409d340, eq_range_arg=true, sorted=true) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:7405
#24 0x0000000004df2b9b in ha_innobase::read_range_first (this=0x7f5eb409d240, start_key=0x7f5eb409d320, end_key=0x7f5eb409d340, eq_range_arg=true, sorted=true) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/handler/ha_innodb.cc:11024
#25 0x0000000003a3d004 in handler::multi_range_read_next (this=0x7f5eb409d240, range_info=0x7f5f74332738) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:6553
#26 0x0000000003a3dfea in DsMrr_impl::dsmrr_next (this=0x7f5eb409d4c8, range_info=0x7f5f74332738) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:6904
#27 0x0000000004e0e400 in ha_innobase::multi_range_read_next (this=0x7f5eb409d240, range_info=0x7f5f74332738) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/innobase/handler/ha_innodb.cc:23701
#28 0x0000000003a3cd36 in handler::ha_multi_range_read_next (this=0x7f5eb409d240, range_info=0x7f5f74332738) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/handler.cc:6491
#29 0x0000000003d4c03f in IndexRangeScanIterator::Read (this=0x7f5eb40a8208) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/range_optimizer/index_range_scan.cc:374
#30 0x000000000377198d in Sql_cmd_update::update_single_table (this=0x7f5eb400f010, thd=0x7f5eb4001040) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_update.cc:876
#31 0x0000000003774aea in Sql_cmd_update::execute_inner (this=0x7f5eb400f010, thd=0x7f5eb4001040) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_update.cc:1824
#32 0x00000000036b6db0 in Sql_cmd_dml::execute (this=0x7f5eb400f010, thd=0x7f5eb4001040) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_select.cc:782
#33 0x000000000362c580 in mysql_execute_command (thd=0x7f5eb4001040, first_level=true) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_parse.cc:3672
#34 0x0000000003631f50 in dispatch_sql_command (thd=0x7f5eb4001040, parser_state=0x7f5f743348e0) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_parse.cc:5406
#35 0x0000000003627f13 in dispatch_command (thd=0x7f5eb4001040, com_data=0x7f5f743359c0, command=COM_QUERY) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_parse.cc:2136
#36 0x0000000003625d5c in do_command (thd=0x7f5eb4001040) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/sql_parse.cc:1465
#37 0x0000000003864f10 in handle_connection (arg=0x1939dd00) at /bv/ushastry/Work/Source/mysql-8.4.5/sql/conn_handler/connection_handler_per_thread.cc:304
#38 0x00000000059e8c0e in pfs_spawn_thread (arg=0x194405c0) at /bv/ushastry/Work/Source/mysql-8.4.5/storage/perfschema/pfs.cc:3061
#39 0x00007f5f7f0bc1aa in start_thread () from /lib64/libc.so.6
#40 0x00007f5f7f141290 in clone3 () from /lib64/libc.so.6

fix trx_cleanup_at_db_startup did not clean up trx_sys->shards (*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: bug118515.patch (application/octet-stream, text), 746 bytes.

Thank you, for the Contribution.

regards,
Umesh