MySQL Bugs: #44327: shutdown scripts for Enterprise Dashboard take too long

Bug #44327	shutdown scripts for Enterprise Dashboard take too long
Submitted:	16 Apr 2009 15:13	Modified:	7 Aug 2009 12:20
Reporter:	Kyle Joiner	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Enterprise Monitor: Server	Severity:	S3 (Non-critical)
Version:	2.0	OS:	Any
Assigned to:	Keith Russell	CPU Architecture:	Any
Tags:	Enterprise Monitor, script, shutdown

Description:
 Initial Description:
This has been reported to the merlin developers but is causing an issue for us as we have to kill -9 the java process to ensure it really dies (and we can restart it).

How to repeat:
Shutdown a highly active Enterprise Dashboard

Suggested fix:
Make the tomcat shutdown script more aggressive.

I was the reporter of this bug. The problem is not minor.

Merlin is not responding properly so I try to restart it.

# date; /opt/mysql/enterprise/monitor/mysqlmonitorctl.sh stop tomcat
Fri Apr 17 08:55:59 CEST 2009
Using CATALINA_BASE:   /opt/mysql/enterprise/monitor/apache-tomcat
Using CATALINA_HOME:   /opt/mysql/enterprise/monitor/apache-tomcat
Using CATALINA_TMPDIR: /opt/mysql/enterprise/monitor/apache-tomcat/temp
Using JRE_HOME:       /opt/mysql/enterprise/monitor/java
Stopping tomcat service ............................................................ [ FAILED ]
# 

In another window I check if the process is still running and wait 5 minutes. After which I kill it because it's still doing something.

Fri Apr 17 09:00:31 CEST 2009
root     17549  252 17.2 1728800 1379568 ?   Sl   Apr15 6634:34 /opt/mysql/enterprise/monitor/java/bin/java -Xmx1024M -Xms1024m -Xss128k -XX:+UseParallelOldGC -XX:+UseBiasedLocking -XX:MaxPermSize=256m -Dxwork.profile.activate=true -Xloggc:gc.log -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file="/opt/mysql/enterprise/monitor/apache-tomcat/conf/logging.properties" -Djava.awt.headless=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/mysql/enterprise/monitor/apache-tomcat/tmp -Xmx768M -Xms768M -Xss128k -XX:+UseParallelOldGC -server -XX:+UseBiasedLocking -Djava.endorsed.dirs=/opt/mysql/enterprise/monitor/apache-tomcat/endorsed -classpath :/opt/mysql/enterprise/monitor/apache-tomcat/bin/bootstrap.jar:/opt/mysql/enterprise/monitor/apache-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/opt/mysql/enterprise/monitor/apache-tomcat -Dcatalina.home=/opt/mysql/enterprise/monitor/apache-tomcat -Djava.io.tmpdir=/opt/mysql/enterprise/monitor/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

This is after running for only 2 days.

If stop fails running restart will also fail.

If you can provide me with information on how to give you more debugging information to determine what Java is doing I'm only too happy to do this.

BTW I don't find this problem minor as it means that we can't just run "mysqlmonitorctl.sh restart" and expect it to work. It doesn't.

Also doing a stop , waiting and doing a start will fail for the same reason as the stop failed. So I consider this a problem.

We've got some fixes in the pipeline for 2.1 to take care of the CPU usage issues, which should remove some of the problem. *However*, the shutdown portion of the script, it makes sense for the script to try "nice" to stop tomcat, but if that fails, kill -9 it.

Patch sent to Keith. We have implemented the solution suggested by Mark Matthews. First the "mysqlmonitorctl.sh" script tries to stop tomcat using the "apache-tomcat/bin/shutdown.sh" script. If the process continues running, the script executes a "kill -9 $PID" to stop the process and check afterward whether it is still running.

Patch installed in versions => 2.1.0.1036.

verified the following section was added to mysqlmonitorctl.sh in build 2.1.0.1036:

>             get_tomcat_pid
>             kill -9 $TOMCAT_PID
>             RUNNING=1
>             COUNTER=60
>             while [ $RUNNING -ne 0 ] && [ $COUNTER -ne 0 ]; do
>                 COUNTER=`expr $COUNTER - 1`
>                 sleep 2
>                 echo $echo_n ".$echo_c"
>                 is_tomcat_running
>                 RUNNING=$?
>             done
>             if [ $RUNNING -eq 0 ]; then
>                 echo " [ OK ]"
>             else
>                 echo " [ FAILED ]"
>             fi

setting back to verified until fixed in 2.0 branch

Waiting to be verified in the next 2.0 build.

Patch installed in versions => 2.0.6.7159.

Bill Weber writes: 
verified section was added in build 2.0.6.7159

An entry was added to the 2.0.6 changelog:

When running the shutdown script for Enterprise Dashboard, it took an unacceptably long time for Tomcat to terminate completely, the java process eventually had to be killed manually.