MySQL Bugs: #41438: Tomcat java process uses too much memory

Bug #41438	Tomcat java process uses too much memory
Submitted:	12 Dec 2008 15:54	Modified:	21 Jul 2009 13:36
Reporter:	Diego Medina	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Enterprise Monitor: Server	Severity:	S2 (Serious)
Version:	2.0.1.7125	OS:	Any (java 1.6)
Assigned to:	MC Brown	CPU Architecture:	Any

Description:
Running the Service manager on Mac OS X for at least 16 hours causes the java process to use 2.08GB of memory.

How to repeat:
1- Install and start the service manager and agent
2- Monitor 2 mysqld instances
3- wait to see how the memory usage goes up beyond the value of -Xmx

System details:

$ /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java -version
java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06-153)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_07-b06-57, mixed mode)

$ ps auxwww | grep "PID\|java" | grep -v grep
USER       PID %CPU %MEM      VSZ    RSS   TT  STAT STARTED      TIME COMMAND
wizard   29242   3.1 51.9  1017756 2177408 s008  S    11:38PM  45:44.19 /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java -Xmx512m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file="/Applications/mysql/enterprise/monitor-2.0.1.7125/apache-tomcat/conf/logging.properties" -Djava.awt.headless=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Applications/mysql/enterprise/monitor-2.0.1.7125/apache-tomcat/tmp -Xmx568M -Xms568M -Xss128k -XX:+UseParallelOldGC -server -Djava.endorsed.dirs=/Applications/mysql/enterprise/monitor-2.0.1.7125/apache-tomcat/endorsed -classpath :/Applications/mysql/enterprise/monitor-2.0.1.7125/apache-tomcat/bin/bootstrap.jar:/Applications/mysql/enterprise/monitor-2.0.1.7125/apache-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/Applications/mysql/enterprise/monitor-2.0.1.7125/apache-tomcat -Dcatalina.home=/Applications/mysql/enterprise/monitor-2.0.1.7125/apache-tomcat -Djava.io.tmpdir=/Applications/mysql/enterprise/monitor-2.0.1.7125/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
wizard   28402   0.0  0.5   343124  20908 s006  S     5:06PM   0:39.80 /usr/bin/java -jar /Users/wizard/Desktop/work/src/etools/mysql-monitor/local-trunk//test/lib/selenium/selenium-server.jar

$ uname -a
Darwin diego-medinas-macbook-pro.local 9.5.0 Darwin Kernel Version 9.5.0: Wed Sep  3 11:29:43 PDT 2008; root:xnu-1228.7.58~1/RELEASE_I386 i386 i386

The logs do not have any entry related to memory errors

Going to the Dashboard->Settings->Product info I see

JVM Max
545.81 MB

JVM Total
545.81 MB

It respects the values from -Xmx

Both agents are using ssl

Running the service manager on the same laptop but using java 1.5 does not show this problem.

We limit heap to 768 MB, how is it using 2 GBs? It probably is related to SSL, but... 

What happens without using SSL?
And then look at Apple bug db to see if java memory usage is a known problem...

We should also take a look at the process parameter list and make sure we are not specifying multiple memory settings.  This was discovered on one of the support servers recently and had the same behavior.  Possibly the 1.6 JVM ignores the memory setting if multiple are set whereas the 1.5 JVM just took the first.

We pass twice the -Xmx parameters, I'll start the service manager using just one of them and see if I can see the same memory problem.

I'm seeing this on a linux 64bit server as well. Uploading a file with some memory stats from the OS on the Java process.

passing only one -Xmx value does not prevent this problem

Has this been resolved by the recent performance/memory work Oldag has been concentrating on?

Verified fixed in 2.1.0.1039

An entry was added to the 2.1.0 changelog:

Running the Service Manager on Mac OS X and monitoring two servers with two agents for at least 16 hours caused the Java process to use 2.08 GB of memory.