Bug #52495 MEM Monitor stack trace - session problem

• Submitted: 31 Mar 2010 9:44
• Modified: 9 Jan 2015 14:42
• Reporter: Roger David Nay
• Status: Won't fix
• Category: MySQL Enterprise Monitor: Server
• Severity: S3 (Non-critical)
• Version: 2.1.2.1158, 2.1.1.1141
• OS: Any
• Assigned to: Assigned Account
• CPU Architecture: Any

[31 Mar 2010 9:44] Roger David Nay

Description:
Following stack trace received with MEM 2.1.2.1158. Dashboard recovers, only to stack trace again intermittently once a day or so.

MySQL Enterprise Monitor "mymonitor.somewhere.com" has encountered a non-recoverable internal error. The Java stack trace follows:

org.hibernate.SessionException: Session is closed!
at org.hibernate.impl.AbstractSessionImpl.errorIfClosed(AbstractSessionImpl.java:49)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1319)
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.hibernate.context.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:301)
at $Proxy8.beginTransaction(Unknown Source)
at com.mysql.etools.monitor.pom.hib.HibernateExecutor.asTransaction(HibernateExecutor.java:183)
at com.mysql.etools.monitor.pom.hib.SchemaMaintenanceExecutor.asTransaction(SchemaMaintenanceExecutor.java:194)
at com.mysql.etools.monitor.pom.hib.HibPersistence.asTransaction(HibPersistence.java:510)
at com.mysql.etools.monitor.pom.hib.HibType.findAttribute(HibType.java:259)
at com.mysql.etools.monitor.pom.hib.HibType.getAttribute(HibType.java:221)
at com.mysql.etools.monitor.pom.hib.HibInventoryObject.updateAttribute(HibInventoryObject.java:267)
at com.mysql.etools.monitor.pom.hib.HibDcNgReaderWriter.saveAttribute(HibDcNgReaderWriter.java:1539)
at com.mysql.etools.monitor.bo.Agent.save(Agent.java:454)
at com.mysql.etools.monitor.bo.Agent.lockedSaveReachable(Agent.java:486)
at com.mysql.etools.monitor.bo.Agent.access$000(Agent.java:74)
at com.mysql.etools.monitor.bo.Agent$4.run(Agent.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at com.mysql.util.LockingExecutor.execute(LockingExecutor.java:39)
at com.mysql.util.LockingExecutor.execute(LockingExecutor.java:33)
at com.mysql.etools.monitor.bo.Agent.saveReachable(Agent.java:458)
at com.mysql.etools.monitor.bo.Agent.lockedAcceptHeartbeat(Agent.java:406)
at com.mysql.etools.monitor.bo.Agent.lockedAcceptHeartbeat(Agent.java:371)
at com.mysql.etools.monitor.bo.Agent.access$100(Agent.java:74)
at com.mysql.etools.monitor.bo.Agent$2.call(Agent.java:360)
at com.mysql.etools.monitor.bo.Agent$2.call(Agent.java:359)
at com.mysql.util.LockingExecutor.execute(LockingExecutor.java:39)
at com.mysql.etools.monitor.bo.Agent.acceptHeartbeat(Agent.java:358)
at com.mysql.etools.monitor.bo.HeartbeatAcceptor.acceptHeartbeat(HeartbeatAcceptor.java:60)
at com.mysql.etools.monitor.bo.HeartBeatCommandProcessor.processRequest(HeartBeatCommandProcessor.java:102)
at com.mysql.merlin.server.MerlinServlet$1.processRequest(MerlinServlet.java:328)
at com.mysql.merlin.server.agent.HeartbeatServlet$1.processRequest(HeartbeatServlet.java:62)
at com.mysql.merlin.server.MerlinServlet.doCommon(MerlinServlet.java:297)
at com.mysql.merlin.server.MerlinServlet.doPost(MerlinServlet.java:419)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.mysql.util.RequestCounterFilter.doFilter(RequestCounterFilter.java:133)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.mysql.etools.monitor.rest.AuthFilter.doFilter(AuthFilter.java:98)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:852)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:584)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1508)
at java.lang.Thread.run(Unknown Source)

How to repeat:
N/A

Suggested fix:
No stack trace.

[29 Apr 2010 5:49] Simon Mudd

Just a note: this error repeats now every 30-50 minutes though sometimes the interval between notifications is a little longer, so it's far more disturbing if only because of the repeated number of mails it generates.

This morning it sent out messages at: 00:00, 00:30, 01:10, 01:40, 02:15, 02:55, 03:35, 04:15, 04:55, 06:00, 06:35, 07:25.
Note the generation of these messages appears to be on exact minute intervals which are multiples of 5. I guess that is not a co-incidence. Looking back at the mails we have received over the last few days this behaviour seems consistent.

Other Warning alerts from merlin come at "odd" minute intervals, that is there is no specific time when they appear.

[17 May 2010 16:08] Enterprise Tools JIRA Robot

Mark Matthews writes: 
We've solved the issue of this sending out a cry-for-help notification when it really doesn't seem to be a critical problem, the application recovers from this. We're leaving this issue open in case others run into the session closed issue, to try and narrow down why it's happening, however.