Description:
Group Replication 多个内存池的 total_allocated 持续增长,
与 current_allocated 差距悬殊(最高 44772x),
内存释放后未归还 OS,长期运行导致服务器可用内存耗尽。
==================== 问题 ====================
Group Replication 多个内存池的 total_allocated 持续增长,
与 current_allocated 差距悬殊(最高 44772x),
内存释放后未归还 OS,长期运行导致服务器可用内存耗尽。
==================== 证据 1:全局内存池 ====================
memory/group_rpl/GCS_XCom::xcom_cache:
COUNT_ALLOC = 57,426,434 COUNT_FREE = 55,808,692
SUM_ALLOC = 46,287,558,421 bytes (≈ 46.3 GiB)
SUM_FREE = 45,213,816,712 bytes (≈ 45.2 GiB)
CURRENT = 1,073,741,709 bytes (≈ 1.07 GiB) ← 差距 43x
HIGH = 1,081,043,071 bytes (≈ 1.08 GiB)
memory/group_rpl/Gcs_message_data::m_buffer:
SUM_ALLOC = 61,576,481,351 bytes (≈ 61.6 GiB)
SUM_FREE = 61,576,481,351 bytes (= 100% 已归还内存池)
CURRENT = 0
HIGH = 30,939,811 bytes
memory/group_rpl/transaction_data:
SUM_ALLOC = 65,210,951,776 bytes (≈ 65.2 GiB)
SUM_FREE = 65,210,951,776 bytes (= 100% 已归还内存池)
CURRENT = 0
HIGH = 33,433,584 bytes
memory/group_rpl/write_set_encoded:
SUM_ALLOC = 5,222,172,915 bytes (≈ 5.22 GiB)
SUM_FREE = 5,222,172,915 bytes (= 100% 已归还内存池)
CURRENT = 0
HIGH = 56,943,405 bytes
memory/group_rpl/certification_info:
SUM_ALLOC = 4,891,567,608 bytes (≈ 4.89 GiB)
SUM_FREE = 4,875,565,184 bytes
CURRENT = 16,002,424 bytes (≈ 16 MiB) ← 差距 305x
HIGH = 135,352,248 bytes
==================== 证据 2:线程级内存分配 ====================
thread_id=68 group_rpl/THD_Gcs_xcom_engine:
current_count_used = 20,259,808 total_alloc = 28.56 GiB (28.55 GiB 在用)
thread_id=78 group_rpl/THD_Gcs_xcom_control::m_xcom_thread:
current_count_used = 23,049 total_alloc = 23.85 GiB (仅 2.2 MiB 在用) ← 极端
thread_id=70 group_rpl/THD_certifier_broadcast:
current_count_used = 161,390 total_alloc = 2.48 GiB
thread_id=57 innodb/clone_gtid_thread:
current_count_used = 3,971 total_alloc = 6.65 GiB (仅 1.27 MiB 在用)
thread_id=58 innodb/srv_purge_thread:
current_count_used = 44 total_alloc = 22.88 GiB ← 差距 4624x
thread_id=59 innodb/srv_worker_thread:
current_count_used = 42 total_alloc = 287.37 GiB ← 差距 44772x ★最极端
==================== 关键发现 ====================
1. COUNT_ALLOC ≈ COUNT_FREE:内存确实被释放了
2. CURRENT 远小于 SUM_ALLOC:释放的内存未归还 OS,仅归还进程内内存池
3. 多个线程 total_allocated 增长与 current 使用量无关,说明内存池分配与实际工作负载分离
4. innodb/srv_worker_thread (59) 的 44772x 差距极为异常,远超已知 "memory pool works as designed" 范围
==================== 根因分析 ====================
COUNT_ALLOC ≈ COUNT_FREE,说明内存分配/释放循环正常。
但 SUM_ALLOC - SUM_FREE = total_allocated 持续增长,
说明 free() 后内存未归还 OS(未调用 munmap/sbrk),
仅保留在 MySQL 进程内内存池中供后续分配复用。
这导致:
- free 命令显示 MySQL 进程 RSS 内存持续增长
- 服务器可用内存逐渐耗尽
- performance_schema 的 total_allocated 数值极具误导性
(实际使用量远小于该值)
How to repeat:
1. 启动 MySQL 8.4.9 Group Replication 集群(3 节点 Single-Primary)
2. 运行一段时间后(约数天)观察:
SELECT * FROM performance_schema.memory_summary_global_by_event_name
WHERE EVENT_NAME LIKE '%group_rpl%';
3. 对比 OS 层面 free -m / ps aux | grep mysqld 确认 RSS 增长
==================== 预期行为 ====================
内存释放后应归还 OS,RSS 应相应下降,
total_allocated 应在稳定工作负载下趋于平稳。
==================== 实际行为 ====================
RSS 持续增长,total_allocated 无限增长,
即使集群处于 idle 状态(无实际事务)也不收缩。
Suggested fix:
N/A