Bug #91690 | MySQL Slave server consume all memory and swap compare to other slave and master | ||
---|---|---|---|
Submitted: | 18 Jul 2018 3:18 | Modified: | 21 Jul 2018 1:52 |
Reporter: | Joey Aldrin Cruz | Email Updates: | |
Status: | Unsupported | Impact on me: | |
Category: | MySQL Server | Severity: | S2 (Serious) |
Version: | 5.6.11 | OS: | CentOS (6.9 Final) |
Assigned to: | MySQL Verification Team | CPU Architecture: | x86 (any) |
Tags: | Memory, MySQL, slave, swap |
[18 Jul 2018 3:18]
Joey Aldrin Cruz
[18 Jul 2018 3:30]
Joey Aldrin Cruz
htop_result
Attachment: htop_result.PNG (image/png, text), 109.92 KiB.
[18 Jul 2018 3:30]
Joey Aldrin Cruz
global_status_result
Attachment: global status.txt (text/plain), 40.58 KiB.
[18 Jul 2018 3:30]
Joey Aldrin Cruz
global_variables_result
Attachment: global_variables.txt (text/plain), 52.23 KiB.
[19 Jul 2018 8:29]
MySQL Verification Team
Hi, please upgrade to a latest 5.6 at least as since 5.6.11 really a lot of bugs are fixed, some of them related to memory usage; especially if you are using stored procedures. best regards Bogdan
[19 Jul 2018 9:28]
Joey Aldrin Cruz
Hello Bogdan, I really appreciate your feedback. But, as I've mentioned. All of the four servers are using the same exact version, configuration, disk size and RAM size, but why only this server experiencing this? I will post current SWAP usage from top command.
[19 Jul 2018 9:29]
Joey Aldrin Cruz
top_result_current
Attachment: top_current_swap.PNG (image/png, text), 15.43 KiB.
[19 Jul 2018 10:18]
Joey Aldrin Cruz
current processlist while on 120GB of swap and 97.7% of memory usage
Attachment: current_processlist.txt (text/plain), 1.54 KiB.
[19 Jul 2018 10:59]
MySQL Verification Team
Hi, > All of the four servers are using the same exact version, > configuration, disk size and RAM size, but why only this > server experiencing this? So yes, it's a bug. Since 5.6.11 there is 29 version releases each with a number of bug fixes, number of them with memory related bugs. Just the next release, 5.6.12, has 3 memory leak bugs fixed (Bug #16754776, #16737332, 14685362). Guessing which one of the bugs you might be hitting is pretty futile waste of time as in any case the only solution is to upgrade in order to solve a problem, so when you are at it, upgrade to latest 5.6; or consider 5.7 kind regards Bogdan
[20 Jul 2018 15:10]
Joey Aldrin Cruz
Hello, I've conducted more research and found out that even on much higher version there's a memory leak issues. And please enlighten us, because why we don't experiencing this on other 12 servers that have the same exact configuration, MySQL version and RAM,DISK size. To be honest, we really want to find the root cause. Is there any connection with waiting for event coordinator state? And if yes; why on the other servers we have the same, but don't have this kind of issues. Our memory is really big enough if you will try to noticed, but still consumed and the swap 120GB, (which is too high to consume) and with only 8 proces base on the processlist. I hope I can hear more strong and convincing reason on this. Cheers, Joey Cruz
[21 Jul 2018 1:52]
Joey Aldrin Cruz
Hello, I'm suspecting that it is because of the slave_parallel_workers which is set to > 1. We will change the value and restart the MySQL and if ever that everything goes well, I'll update this bug report, so that whoever who can read this, they can have an idea on how can they fix it. Cheers,