Bug #97293 | group replication plugin memory leak | ||
---|---|---|---|
Submitted: | 18 Oct 2019 16:51 | Modified: | 20 Nov 2019 16:46 |
Reporter: | Benoît Guyard | Email Updates: | |
Status: | Not a Bug | Impact on me: | |
Category: | MySQL Server: Group Replication | Severity: | S2 (Serious) |
Version: | at least 8.0.16, 8.0.17 & 8.0.18 | OS: | Ubuntu (Ubuntu 18.04.2 LTS) |
Assigned to: | MySQL Verification Team | CPU Architecture: | x86 (GNU/Linux 4.15.0 x86_64) |
Tags: | group replication, Leak, Memory |
[18 Oct 2019 16:51]
Benoît Guyard
[21 Oct 2019 17:26]
MySQL Verification Team
Thanks for the report, I can't find anything with valgrind but I'll leave it running now for a few days with your script & config to see if I can reproduce this behavior.
[25 Oct 2019 10:16]
MySQL Verification Team
Hi, four days later and I still don't see any ram issues with 8.0.18
[30 Oct 2019 17:02]
Benoît Guyard
Hello Bogdan, indeed, it seems i've been jumping to conclusions a bit hastily: looking more closely at monitoring metrics, the mem leak behavior seems to have actually started after an upgrade of libssl & openssl pkgs, and not when this group replication cluster got upgraded from 5.7 to 8.0.16 (which happened before). libssl/openssl package upgrades i'm referring too is: > Upgrade: libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.3, 1.1.1-1ubuntu2.1~18.04.4) > Upgrade: openssl:amd64 (1.1.1-1ubuntu2.1~18.04.3, 1.1.1-1ubuntu2.1~18.04.4) Then the cluster got upgraded to 8.0.17 and the mem leak issue was still happening, but i had not started digging deeper at this stage yet and was just witnessing that mysql was still eating ram up until exhaustion. Then the cluster got upgraded to 8.0.18 and upon still witnessing the same behavior i started digging.. but did not wait long enough before filing in this bug report: it turns out that with v8.0.18, mysql mem usage actually gets stable at some point and also, the group replication xcom cache only grows until it reaches 1G, exactly as it is documented there: https://dev.mysql.com/doc/refman/8.0/en/group-replication-performance-xcom-cache.html The script i was using to monitor this shows it very clearly: Tue Oct 29 16:07:22 UTC 2019 - memory/group_rpl current_alloc: 968.37 MiB - total alloc: 2.40 GiB [..] Tue Oct 29 20:37:24 UTC 2019 - memory/group_rpl current_alloc: 1000.91 MiB - total alloc: 2.43 GiB [..] Tue Oct 29 23:57:24 UTC 2019 - memory/group_rpl current_alloc: 1024.00 MiB - total alloc: 2.46 GiB [..] Wed Oct 30 16:07:27 UTC 2019 - memory/group_rpl current_alloc: 1024.00 MiB - total alloc: 2.46 GiB Bottom line is that i still don't really understand what was making mysql so unhappy before, but somehow it seems that it was related to libssl/openssl & mysql versions < v8.0.18. Sorry for the noise and thank you for your time! Benoît
[4 Nov 2019 14:46]
MySQL Verification Team
thanks for the update!