Bug #93730 Memory Leak
Submitted: 23 Dec 2018 12:39 Modified: 26 Feb 8:56
Reporter: Frederic Steinfels Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server Severity:S5 (Performance)
Version:8.0.13 OS:Fedora (4.20.6-200.fc29.x86_64)
Assigned to: Bogdan Kecman CPU Architecture:x86
Tags: memory leak

[23 Dec 2018 12:39] Frederic Steinfels
Description:
Mysql is grabbing memory at an alarming rate of about 20 gigabytes per day until the server becomes so slow that I have to restart mysqld.

How to repeat:
Please advise what details you need.
[23 Dec 2018 12:40] Frederic Steinfels
Memory Map

Attachment: mem.txt (text/plain), 40.99 KiB.

[24 Dec 2018 1:58] zhai weixiang
If you are using mysql8.0, you can get some information of how memory is used by quering tables in performance_schema. 

https://dev.mysql.com/doc/refman/8.0/en/memory-summary-tables.html
[24 Dec 2018 19:20] Bogdan Kecman
Hi Frederic,

> Please advise what details you need.

We need *all* details you can provide taken that we can't reproduce this nor this is something anyone else reported.

So for start
 - config files
 - system details (os, kernel ..)
 - memory usage details from SYS table ( https://dev.mysql.com/doc/refman/8.0/en/sys-schema-object-index.html look at all *memory* views and get us select * from them)

Furthermore, 
 - how many connections do you have? 
 - Are your number of connections rising or ? 
 - Does your apps close connections explicitly they wait for timeout?

And finally. If you close all apps, and close all connections to the database, is the ram usage same or it starts dropping?

Thanks
Bogdan
[15 Jan 15:06] Frederic Steinfels
I figured out that mysql is accumulating up to 200gb of memory and then freeing it again going back to 50gb. I am currently investigating this. I have to retract my claim this is a memory loss. So you might consider closing this and I will reopen when I have all the data ready.
[15 Jan 17:38] Bogdan Kecman
Thanks for the update,

closed as "not a bug" for now, but you can always reopen (or open new one)

all best
Bogdan
[19 Feb 16:19] Frederic Steinfels
I have uploaded extensive log files and the config. It might very well be a config problem combined with something else that is going on. However it might also be some other problem worth fixing. As I am under the impression this problem only occured since updating to 8.x.
[22 Feb 8:25] Frederic Steinfels
Hi

Please reopen the bug as you proposed earlier. 

Tonight it seems mysqld has used so much memory that it ran out and got killed. I will upload the core file and log files for inspection. 

Regards
[22 Feb 8:31] Frederic Steinfels
Please reopen that bug as discussed.
[22 Feb 12:04] Bogdan Kecman
Hi,

Thanks for the additional data (you can delete the core now).
Will let you know when we get anywhere with it

thanks
Bogdan
[22 Feb 16:49] Frederic Steinfels
It might be that the crash that I reported due to too much memory usage might have been my fault. I fixed an error in an sql statement. Will report again if that happens. However the fact that VIRT (under htop) will go up to to 250 GB sometimes and then falls again is still there. Might also be a config error though but I am not sure - I think its worth looking into.
[25 Feb 9:08] Bogdan Kecman
Hi,

Yes, returning to not-a-bug, your config might need some changes, number of memory allocations there are "per connection" so with a lot of connections you can easily go up to 250G. I can suggest contacting support team wrt optimization of the whole system and possibly optimization of those queries you are having issues with.

all best
Bogdan
[26 Feb 8:56] Frederic Steinfels
I found one SQL statement that took HOURS to complete and therefore this backlog of connections built up. The memory / channel stuff was recommended by phpmyadmin and others, will also try changing them if necessary. I assume this was all my fault so sorry for bothering you on this.
[26 Feb 10:19] Bogdan Kecman
No worries. I can suggest you consider MySQL Support, it helps with issues like this, both to quickly solve and to prevent + with Enterprise plan you get to use MySQL Enterprise Monitor that would detect issues like this and alert you even before you experience a problem.

all best
Bogdan