Bug #55612 Excessive agent memory usage on RHEL5 X86_64
Submitted: 28 Jul 2010 16:58 Modified: 9 Nov 2015 15:25
Reporter: Shannon Wade Email Updates:
Status: Won't fix Impact on me:
None 
Category:MySQL Enterprise Monitor: Agent Severity:S2 (Serious)
Version:2.2.2.1729 OS:Linux (RH5)
Assigned to: Jan Kneschke CPU Architecture:Any

[28 Jul 2010 16:58] Shannon Wade
Description:
2.2.2.1729 Agent install on RHEL5 X86_64 without quan or proxy or SSL enabled uses 192-256M per instance.

# Version: 2.2.2.1729

[mysql-proxy]

# Common Parameters
plugins=agent
keepalive = true
log-file = mysql-monitor-agent.log
pid-file=/users/swade/enterprise/agent/mysql-monitor-agent.pid

# Agent Parameters
agent-mgmt-hostname = http://127.0.0.1:18001/heartbeat
agent-mgmt-username = agent
agent-mgmt-password = blah
mysqld-instance-dir= etc/instances
agent-item-files = share/mysql-monitor-agent/items/items-mysql-monitor.xml,share/mysql-monitor-agent/items/custom.xml
agent-uuid = e906de47-138f-46cc-a89f-2ee60ae93425

# Proxy Parameters
#proxy-address=:6446
#proxy-backend-addresses = /users/swade/mysql.sock
#proxy-lua-script = lib/mysql-monitor-agent/lua/quan.lua

How to repeat:
Plain MEM and agent install (no quan nor proxy enabled) on 2.6.18-8.1.1.el5

29160 swade     21   0 48084 3272 2596 S    0  0.1   0:00.12 mysql-monitor-a                   
29167 swade     15   0  256m 5664 2516 S    0  0.1   0:00.11 mysql-monitor-a  

after restart agent and monitor:

 2079 swade     15   0  192m 4372 2276 S    0  0.1   0:00.05 mysql-monitor-a     

Generated a new uuid and restarted agent:

11405 swade     15   0  257m 5768 2516 S    0  0.1   0:00.11 mysql-monitor-a   

agent log blank except normal connect to monitor and db

Suggested fix:
Agent memory usage is normally much less, so seems something is odd here on this platform.
[4 Aug 2010 15:19] Enterprise Tools JIRA Robot
Jan Kneschke writes: 
Please run valgrind with:

{noformat}
$ G_SLICE="always-malloc" \
    G_DEBUG="resident-modules" \
    MYSQL_PROXY_WRAPPER="valgrind --trace-children=yes --leak-check=full" \
    ./bin/mysql-monitor-agent --defaults-file=etc/mysql-monitor-agent.ini
{noformat}

The G_SLICE="always-malloc" will help valgrind to remove some false positives, the G_DEBUG will remove most of the ??? as it doesn't unloaded the plugins (but it will some some leaks that we can easily filter out).

see http://library.gnome.org/devel/glib/stable/glib-running.html
[5 Aug 2010 2:22] MySQL Verification Team
Hi jan,

-bash-3.1$ echo $G_SLICE
always-malloc
-bash-3.1$ echo $G_DEBUG
resident-modules
-bash-3.1$ echo $MYSQL_PROXY_WRAPPER
valgrind --trace-children=yes --leak-check=full
-bash-3.1$ valgrind ./bin/mysql-monitor-agent --defaults-file=etc/mysql-monitor-agent.ini 

==4481== Memcheck, a memory error detector.
==4481== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al.
==4481== Using LibVEX rev 1658, a library for dynamic binary translation.
==4481== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.
==4481== Using valgrind-3.2.1, a dynamic binary instrumentation framework.
==4481== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al.
==4481== For more details, rerun with: -v
==4481== 
==4485== 
==4485== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 1)
==4485== malloc/free: in use at exit: 17,496 bytes in 427 blocks.
==4485== malloc/free: 630 allocs, 203 frees, 33,268 bytes allocated.
==4485== For counts of detected errors, rerun with: -v
==4485== searching for pointers to 427 not-freed blocks.
==4485== checked 177,120 bytes.
==4485== 
==4485== LEAK SUMMARY:
==4485==    definitely lost: 26 bytes in 1 blocks.
==4485==      possibly lost: 0 bytes in 0 blocks.
==4485==    still reachable: 17,470 bytes in 426 blocks.
==4485==         suppressed: 0 bytes in 0 blocks.
==4485== Use --leak-check=full to see details of leaked memory.
==4484== 
==4484== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 1)
==4484== malloc/free: in use at exit: 16,532 bytes in 397 blocks.
==4484== malloc/free: 587 allocs, 190 frees, 32,313 bytes allocated.
==4484== For counts of detected errors, rerun with: -v
==4484== searching for pointers to 397 not-freed blocks.
==4484== checked 176,232 bytes.
==4484== 
==4484== LEAK SUMMARY:
==4484==    definitely lost: 0 bytes in 0 blocks.
==4484==      possibly lost: 0 bytes in 0 blocks.
==4484==    still reachable: 16,532 bytes in 397 blocks.
==4484==         suppressed: 0 bytes in 0 blocks.
==4484== Reachable blocks (those to which a pointer was found) are not shown.
==4484== To see them, rerun with: --show-reachable=yes

During that top & aux reports:

 4782 swade     15   0  256m 8024 3888 S    0  0.2   0:01.79 mysql-monitor-a  

-bash-3.1$ ps aux |grep agent
swade     4782  0.1  0.1 262756  8024 pts/1    Sl+  04:00   0:01 /users/swade/enterprise/agent/libexec/mysql-monitor-agent --defaults-file=etc/mysql-monitor-agent.ini
[29 Feb 2012 22:28] MySQL Verification Team
Are you sure you didn't want to use a heap profiler to find out where/why the memory usage is high while it's running (but not necessarily leaked after shutdown).

http://google-perftools.googlecode.com/svn/trunk/doc/heapprofile.html
[9 Nov 2015 15:25] Mark Leith
Not an issue in 3.x.