Bug #108261 Contribution by Tencent: start may stuck if too many external prepared trxs
Submitted: 24 Aug 2022 2:43 Modified: 24 Aug 2022 6:23
Reporter: yewei Xu (OCA) Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: XA transactions Severity:S2 (Serious)
Version:5.7,8.0 OS:Any
Assigned to: CPU Architecture:Any
Tags: Contribution

[24 Aug 2022 2:43] yewei Xu
Description:
Number of slots in xid_list passed to trx_recover_for_mysql is allocated default to MAX_XID_LIST_SIZE, but if allocate failed, this will reduce by half till MIN_XID_LIST_SIZE(128), if we have more than MIN_XID_LIST_SIZE external prepared trxs, start will get stuck.
This is because in xa::recovery::recover_one_ht we only break the while loop when got < info->len, but got will always == info->len if all trxs in info->list is external prepared trx.

How to repeat:
diff --git a/sql/xa.cc b/sql/xa.cc
index 63491b4a464..25f76f69ca0 100644
--- a/sql/xa.cc
+++ b/sql/xa.cc
@@ -311,6 +311,7 @@ int ha_recover(Xid_commit_list *commit_list, Xa_state_list *xa_list) {
 
   for (info.len = MAX_XID_LIST_SIZE;
        info.list == nullptr && info.len > MIN_XID_LIST_SIZE; info.len /= 2) {
+    DBUG_EXECUTE_IF("simulate_low_memory", { info.len = MIN_XID_LIST_SIZE; });
     info.list = new (std::nothrow) XA_recover_txn[info.len];
   }
   if (!info.list) {

-------------------------------------------------------------------------------

call mtr.add_suppression("\\[Server\\] Found 1000 prepared XA transactions");

--connection default

CREATE TABLE t (a INT) ENGINE=innodb;

--let $prepared_trx_number = 1000
--let $k = 0
--disable_query_log
while ($k < $prepared_trx_number)
{
    --connect(conn1,localhost,root,,test)
    --eval XA START   'prepared_trx_$k'
    --eval INSERT INTO t SET a=$k
    --eval XA END     'prepared_trx_$k'
    --eval XA PREPARE 'prepared_trx_$k'
    --inc $k
    --disconnect conn1
}
--enable_query_log

--connect(conn1,localhost,root,,test)

--let $restart_parameters = restart: --debug=+d,simulate_low_memory
--source include/restart_mysqld.inc

--let $k = 0
--disable_query_log
while ($k < $prepared_trx_number)
{
    --eval XA COMMIT 'prepared_trx_$k'
    --inc $k
}
--enable_query_log

DROP TABLE t;

Suggested fix:
Always allocate MAX_XID_LIST_SIZE,if allocate failed,return ER_SERVER_OUTOFMEMORY
[24 Aug 2022 6:23] MySQL Verification Team
Hello  Yewei Xu,

Thank you for the report and contribution.

regards,
Umesh