Bug #53077 "Invalid UUID" exceptions in Server log when agent has obsolete configuration
Submitted: 22 Apr 2010 18:28 Modified: 26 Apr 2011 11:12
Reporter: Andrii Nikitin Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Enterprise Monitor: Server Severity:S3 (Non-critical)
Version:2.1, 2.2.0.1693 OS:Any
Assigned to: MEM Agent Devs CPU Architecture:Any

[22 Apr 2010 18:28] Andrii Nikitin
Description:
When agent has some obsolete instances in configuration, which are now being monitored by another agent:

1. Instance is shown as crossed out in dashboard, but graphs are updated properly.

2. "Invalid UUID" exceptions are flooded in MEM Server log.

$ grep "Invalid UUID" mysql-monitor.log.* |wc -l
5429

java.lang.IllegalArgumentException: Invalid UUID string:
java.util.UUID.fromString(Unknown Source)
com.mysql.etools.monitor.bo.Agent.processServers(Agent.java:1131)
com.mysql.etools.monitor.bo.Agent.processInventoryType(Agent.java:962)
com.mysql.etools.monitor.bo.Agent.processInventoryTasks(Agent.java:953)
com.mysql.etools.monitor.bo.Agent.processTasks(Agent.java:578)
com.mysql.etools.monitor.bo.Agent.lockedAcceptHeartbeat(Agent.java:417)
com.mysql.etools.monitor.bo.Agent.lockedAcceptHeartbeat(Agent.java:394)
com.mysql.etools.monitor.bo.Agent.access$100(Agent.java:77)
com.mysql.etools.monitor.bo.Agent$3.call(Agent.java:383)
com.mysql.etools.monitor.bo.Agent$3.call(Agent.java:382)
com.mysql.etools.concurrent.LockingExecutor.execute(LockingExecutor.java:39)
com.mysql.etools.monitor.bo.Agent.acceptHeartbeat(Agent.java:381)
com.mysql.etools.monitor.bo.HeartbeatAcceptor.acceptHeartbeat(HeartbeatAcceptor.java:60)
com.mysql.etools.monitor.bo.HeartBeatCommandProcessor.processRequest(HeartBeatCommandProcessor.java:102)
com.mysql.merlin.server.MerlinServlet$1.processRequest(MerlinServlet.java:328)
com.mysql.merlin.server.agent.HeartbeatServlet$1.processRequest(HeartbeatServlet.java:62)
com.mysql.merlin.server.MerlinServlet.doCommon(MerlinServlet.java:297)
com.mysql.merlin.server.MerlinServlet.doPost(MerlinServlet.java:419)
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
com.mysql.merlin.server.RequestCounterFilter.doFilter(RequestCounterFilter.java:117)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
com.mysql.etools.monitor.rest.AuthFilter.doFilter(AuthFilter.java:121)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

How to repeat:
This situation may be observed when agent was monitoring several MySQL instances and later at least one instance is moved to other box (and is monitored by another agent).

Here are steps how to reproduce the problem:

1. Configure agent to monitor two MySQL instances, let's say on ports 13306 and 3306 and make sure that dashboard shows them properly.

2. Shutdown server on 3306 and start it in new location, let's say on port 3307.

3. Configure second agent instance to monitor MySQL instance on port 3307.

4. Wait couple minutes and observe that instance in MySQL dashboard is crossed out but its graphs are being updated properly

5. Restart tomcat

6. After a while mysql-monitor.log is full of messages "Invalid UUID String" with stacktrace 

Suggested fix:
1. Clear error message about obsolete instance in agent configuration instead of "Invalid UUID String"
2. Do not flood log with identical stacktraces
[23 Mar 2011 16:06] Jan Kneschke
After the MEM-server is restarted, the agent sends:

   <taskId>9223372036854775799</taskId>
   <command>list_instances</command>
   <utc>2011-03-23T15:45:45.074Z</utc>
   <types>
    <type>
     <namespace>mysql</namespace>
     <classname>server</classname>
     <instances>
      <em2_instance>
       <instancename>0e982db4-2577-4fc0-b4d2-d1721748bfbf</instancename>
       <parent>
        <namespace>os</namespace>
        <classname>host</classname>
        <instancename>ssh:{e4:43:f1:5e:87:93:04:5c:df:30:88:35:71:98:f0:2d}</instancename>
       </parent>
      </em2_instance>
      <em2_instance>
       <instancename></instancename>
      </em2_instance>
     </instances>
    </type>
   </types>
  </task>

... a instance of a MySQL server with a empty instance name. That doesn't happen on normal startup.
[26 Apr 2011 11:12] Mark Leith
Fixed within the 2.3.3 release within the Agent:

2020 jan@mysql.com	2011-03-24
     fixes bug(s): http://bugs.mysql.com/53077
     don't add values for mysql::server instances that we don't have a UUID for yet (fixes #53077/EM-4335)