MySQL Bugs: #53880: agent 2.2.1 shows growing Agent Memory Usage (memory leak) on Windows

Bug #53880	agent 2.2.1 shows growing Agent Memory Usage (memory leak) on Windows
Submitted:	21 May 2010 12:22	Modified:	29 Jun 2010 7:44
Reporter:	Carsten Segieth	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Enterprise Monitor: Agent	Severity:	S1 (Critical)
Version:	2.2.1.1717	OS:	Windows
Assigned to:	Michael Schuster	CPU Architecture:	Any

Description:
The Windows agent shows increasing mem usage, constantly growing around ~1MB / h.

For non-Windows systems I could not see such growing.

How to repeat:
It happens both with updated (2.1.2.1166 --> 2.2.1.1717) and fresh installed agents 2.2.1.1717, all are pinging in to the same MEM server.
In case it matters, this server is an updated server (2.1.2.1163 --> 2.2.1.1720).

all Win agents for over 1 1/2 days

Attachment: bug#53880 - agent 2.2.1 mem leak on Win - all Win agents.jpg (image/jpeg, text), 122.43 KiB.

fresh installed 2.2.1.1717 with debug log

Attachment: bug#53880 - agent 2.2.1 mem leak on Win - Win agent started with debug log.jpg (image/jpeg, text), 111.19 KiB.

can be seen also with an agent 2.2.0.1705

Mark Leith writes: 
Could you please check whether all Windows machine in that group are showing the same symptoms? It looks like from the aggregated graph they are not. 

Do you have a longer graph for some agents that see this problem?

If there are differences between the windows machines - what differences are there are on the installs showing this problem?

I have not yet an idea what 'external' condition may cause the leak / growing mem usage:
- the agents do not show all the same behaviour, but most ...
- most are still growing, both against fresh or updated 1720 svc mgr install
- most growing are with self-quan, one without self-quan is growing much slower (not at ~68M after 4 days)
- one with self-quan was shrinking from a max of 220 to now ~40
- one without self-quan and with proxy load is around a 'fix' level of 25M
- max range of data is around 5 days

on WinXP Home: fresh installed agent 2.2.1.1717, monitoring and configured as proxy for MEM on the same host, with QUAN + self-Quan, did not show any growth within ~ 6h. Max agent mem value was ~22M, then shrinking over 2-3 h to ~18M

restarted without self-quan 3 agents of my "usual" setup
- they all showed the mem growth when using self-quan
- now we need to wait some time to see whether this shows us a change

I had running 3 agents 2.1.2.1166 (WinXP, 2003, Vista) for ~12h, reporting to the 2.2.1.1721 svc mgr on qa-merlin and they also showed a mem growth of ~0,5M - 0,7M/h. Starting now 2.1.1.1144 on these systems for comparsion.



Attachment: 10380_agent-mem.png.jpg (image/jpeg, text), 138.43 KiB.

Jan Kneschke writes: 
going through the changelogs of:

    $ bzr log -v -r tag:build_1144..tag:build_1166

there is nothing really windows specific. Can we narrow it down to the builds between those two ?

Bill Weber writes: 
In 2.1.2, this first appeared in build 1166 and was possibly introduced in 1165 although there was no Windows agent. In 2.2.0, this first appeared in 1674 and was possibly introduced in 1668 as there was no Windows agent for that build either. As a note, the SSL fix was first in both 2.1.2.1165 and 2.2.0.1668.

Jan Kneschke writes: 
Using 2.1.2-1166 and using a setup like:

remote (macosx): mysqld (5.5.2) + mem-server (2.2.0-...)
win32: mem-agent 2.1.2-1166

mem-agent -> mysqld: no SSL
mem-agent -> mem-server: no SSL

No memory growth. It stays at 11MByte for the whole time. 

Using the pasted images, we should see a mem-growth jump each 30min, but here it just goes down slightly.

I used the 'perfmon', and tracked the 'Virtual Bytes' and 'Private Bytes' of the object "Process" and the instance "mysql-monitor-agent".

Next is the same test with a local mysqld.

Jan Kneschke writes: 
switching from remote-monitoring a MySQL-server on macosx to a local mysql-server on windows ... and still the same flat line.

Jan Kneschke writes: 
2.1.2-1166 installed from installer without QUAN (aka no proxy plugin)

{noformat}
[mysql-proxy]

keepalive = true
plugins=agent
agent-mgmt-hostname = http://agent:mysql@192.168.2.110:8080/Monitor2/heartbeat
mysqld-instance-dir= etc/instances
agent-item-files = share/mysql-proxy/items/items-mysql-monitor.xml,share/mysql-proxy/items/custom.xml

agent-uuid = 2a0e3058-a175-4391-9987-e2ce39949e63
log-file = mysql-monitor-agent.log
log-level=message
pid-file=C:\\Program Files\\MySQL\\Enterprise\\Agent\\mysql-monitor-agent.pid

basedir=C:\\PROGRA~1\\MySQL\\ENTERP~1\\Agent
{noformat}

is flat at 11.7MByte. Agent and MySQL run in a winxp-VM in virtualbox on MacOS X while the mem-server runs on the Mac itself.

This is listed as affecting win32 agents. Since there is no x64 agent on Windows, I assume this also affects the standard win32 agent running on x64?

Roel, yes, there is only one (32bit) agent for Windows, and the problem can be seen with both running on 32 + 64bit systems.

agent 2.2.2.1726 does not show the mem growth for me

Attachment: bug#53880 - 2.2.2.1726 does not show mem growth for me.jpg (image/jpeg, text), 135.92 KiB.

Michael Schuster writes: 
relevant bzr log:

revno: 1879
committer: michael.schuster@oracle.com
branch nick: agent_winfix_2.2
timestamp: Wed 2010-06-16 00:15:55 -0700
message:
  windows build failure fix (another guess)
------------------------------------------------------------

------------------------------------------------------------
revno: 1877
committer: michael.schuster@oracle.com
branch nick: agent_winfix_2.2
timestamp: Tue 2010-06-15 12:21:25 -0700
message:
  tentative fix for EM-4523 "agent 2.2.1 shows growing Agent Memory Usage (memory leak) on Windows"
  lacking proper Windows devel. env, this is a first guess

Build 2.2.2.1726 doesn't show signif. growth after 1.5 days. Jump at 10.00 h caused by network trouble, happened to all agents

Attachment: bug#53880 - 2.2.2.1726 does not show significant mem growth after 1.5 days.jpg (image/jpeg, text), 131.13 KiB.

also on Bill's test system (which is completly separate from my tests) the agent mem looks fine

Attachment: bug#53880 - 2.2.2.1726 does not show significant mem growth after 1.5 days (Bill's testsystem).jpg (image/jpeg, text), 111.65 KiB.

Andy Bang writes: 
In build 2.2.2.1727.

Carsten Segieth writes: 
OK in 2.2.2.1727

An entry has been added to the 2.2.2 changelog: 

        On Windows platforms, the &merlin_agent; could show increasing                                
        memory usage over time.