Bug #120488 MySQL 8.4 Group Replication memory pools not released to OS srv_worker_thread 44,772x total/current ratio; xcom_cache ho
Submitted: 17 May 11:04
Reporter: 全 郑 Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:8.4.9 OS:Linux (rocky linux 9.7)
Assigned to: CPU Architecture:x86 (9.7)
Tags: memory leak

[17 May 11:04] 全 郑
Description:
Group Replication 多个内存池的 total_allocated 持续增长,
与 current_allocated 差距悬殊(最高 44772x),
内存释放后未归还 OS,长期运行导致服务器可用内存耗尽。

==================== 问题 ====================
Group Replication 多个内存池的 total_allocated 持续增长,
与 current_allocated 差距悬殊(最高 44772x),
内存释放后未归还 OS,长期运行导致服务器可用内存耗尽。

==================== 证据 1:全局内存池 ====================
memory/group_rpl/GCS_XCom::xcom_cache:
  COUNT_ALLOC = 57,426,434   COUNT_FREE = 55,808,692
  SUM_ALLOC   = 46,287,558,421 bytes  (≈ 46.3 GiB)
  SUM_FREE    = 45,213,816,712 bytes  (≈ 45.2 GiB)
  CURRENT     =  1,073,741,709 bytes  (≈  1.07 GiB)  ← 差距 43x
  HIGH        =  1,081,043,071 bytes  (≈  1.08 GiB)

memory/group_rpl/Gcs_message_data::m_buffer:
  SUM_ALLOC   = 61,576,481,351 bytes  (≈ 61.6 GiB)
  SUM_FREE    = 61,576,481,351 bytes  (= 100% 已归还内存池)
  CURRENT     = 0
  HIGH        = 30,939,811 bytes

memory/group_rpl/transaction_data:
  SUM_ALLOC   = 65,210,951,776 bytes  (≈ 65.2 GiB)
  SUM_FREE    = 65,210,951,776 bytes  (= 100% 已归还内存池)
  CURRENT     = 0
  HIGH        = 33,433,584 bytes

memory/group_rpl/write_set_encoded:
  SUM_ALLOC   = 5,222,172,915 bytes   (≈  5.22 GiB)
  SUM_FREE    = 5,222,172,915 bytes   (= 100% 已归还内存池)
  CURRENT     = 0
  HIGH        = 56,943,405 bytes

memory/group_rpl/certification_info:
  SUM_ALLOC   = 4,891,567,608 bytes   (≈  4.89 GiB)
  SUM_FREE    = 4,875,565,184 bytes
  CURRENT     =   16,002,424 bytes    (≈    16 MiB)  ← 差距 305x
  HIGH        =  135,352,248 bytes

==================== 证据 2:线程级内存分配 ====================
thread_id=68  group_rpl/THD_Gcs_xcom_engine:
  current_count_used = 20,259,808   total_alloc = 28.56 GiB  (28.55 GiB 在用)

thread_id=78  group_rpl/THD_Gcs_xcom_control::m_xcom_thread:
  current_count_used =     23,049   total_alloc = 23.85 GiB  (仅 2.2 MiB 在用)  ← 极端

thread_id=70  group_rpl/THD_certifier_broadcast:
  current_count_used =    161,390   total_alloc =  2.48 GiB

thread_id=57  innodb/clone_gtid_thread:
  current_count_used =      3,971   total_alloc =  6.65 GiB  (仅 1.27 MiB 在用)

thread_id=58  innodb/srv_purge_thread:
  current_count_used =         44   total_alloc = 22.88 GiB  ← 差距 4624x

thread_id=59  innodb/srv_worker_thread:
  current_count_used =         42   total_alloc = 287.37 GiB ← 差距 44772x  ★最极端

==================== 关键发现 ====================
1. COUNT_ALLOC ≈ COUNT_FREE:内存确实被释放了
2. CURRENT 远小于 SUM_ALLOC:释放的内存未归还 OS,仅归还进程内内存池
3. 多个线程 total_allocated 增长与 current 使用量无关,说明内存池分配与实际工作负载分离
4. innodb/srv_worker_thread (59) 的 44772x 差距极为异常,远超已知 "memory pool works as designed" 范围

==================== 根因分析 ====================
COUNT_ALLOC ≈ COUNT_FREE,说明内存分配/释放循环正常。
但 SUM_ALLOC - SUM_FREE = total_allocated 持续增长,
说明 free() 后内存未归还 OS(未调用 munmap/sbrk),
仅保留在 MySQL 进程内内存池中供后续分配复用。

这导致:
- free 命令显示 MySQL 进程 RSS 内存持续增长
- 服务器可用内存逐渐耗尽
- performance_schema 的 total_allocated 数值极具误导性
  (实际使用量远小于该值)

How to repeat:
1. 启动 MySQL 8.4.9 Group Replication 集群(3 节点 Single-Primary)
2. 运行一段时间后(约数天)观察:
   SELECT * FROM performance_schema.memory_summary_global_by_event_name
   WHERE EVENT_NAME LIKE '%group_rpl%';
3. 对比 OS 层面 free -m / ps aux | grep mysqld 确认 RSS 增长

==================== 预期行为 ====================
内存释放后应归还 OS,RSS 应相应下降,
total_allocated 应在稳定工作负载下趋于平稳。

==================== 实际行为 ====================
RSS 持续增长,total_allocated 无限增长,
即使集群处于 idle 状态(无实际事务)也不收缩。

Suggested fix:
N/A