Bug #47727 Monitoring data loss, missing graphs, exceptions in log
Submitted: 29 Sep 2009 20:01 Modified: 8 Jun 2010 10:59
Reporter: Andrii Nikitin Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Enterprise Monitor: Agent Severity:S1 (Critical)
Version:2.1.0.1093->2.1.0.1096, 2.2.1.1717 OS:Any
Assigned to: Assigned Account CPU Architecture:Any
Tags: windmill

[29 Sep 2009 20:01] Andrii Nikitin 
Description:
Messages from agent log:

2009-09-16 13:15:59: (critical) G:\bs\bs\merlin\agent-2.0\src\mysql-proxy-0.7.0r1489_20090811_1738\plugins\agent\network-io.c:841: starting task 2 for mysql::server[418fdade-6547-4ccd-9556-c85cedfbc99a]
2009-09-17 01:16:19: (critical) exception received from server: Internal error: known items are required for list_instances processing

Messages from server log:
java.lang.RuntimeException: known items are required for list_instances processing
at com.mysql.etools.monitor.bo.Agent.processInventoryTasks(Agent.java:846)
at com.mysql.etools.monitor.bo.Agent.processTasks(Agent.java:550)
at com.mysql.etools.monitor.bo.Agent.lockedAcceptHeartbeat(Agent.java:394)
at com.mysql.etools.monitor.bo.Agent.lockedAcceptHeartbeat(Agent.java:371)
at com.mysql.etools.monitor.bo.Agent.access$100(Agent.java:74)
at com.mysql.etools.monitor.bo.Agent$2.call(Agent.java:360)
at com.mysql.etools.monitor.bo.Agent$2.call(Agent.java:359)
at com.mysql.util.LockingExecutor.execute(LockingExecutor.java:39)
at com.mysql.etools.monitor.bo.Agent.acceptHeartbeat(Agent.java:358)
at com.mysql.etools.monitor.bo.HeartbeatAcceptor.acceptHeartbeat(HeartbeatAcceptor.java:60)
at com.mysql.etools.monitor.bo.HeartBeatCommandProcessor.processRequest(HeartBeatCommandProcessor.java:102)
at com.mysql.merlin.server.MerlinServlet$1.processRequest(MerlinServlet.java:328)
at com.mysql.merlin.server.agent.HeartbeatServlet$1.processRequest(HeartbeatServlet.java:62)
at com.mysql.merlin.server.MerlinServlet.doCommon(MerlinServlet.java:297)
at com.mysql.merlin.server.MerlinServlet.doPost(MerlinServlet.java:419)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.mysql.util.RequestCounterFilter.doFilter(RequestCounterFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.mysql.etools.monitor.rest.AuthFilter.doFilter(AuthFilter.java:98)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Unknown Source)

2009-09-17 01:16:19,381 ERROR [http-18080-7:com.mysql.infra.com.mysql.merlin.server.MerlinServlet] <?xml version='1.0'?><exceptions><error><![CDATA[Internal error: known items are required for list_instances processing]]></error></exceptions>
java.lang.RuntimeException: known items are required for list_instances processing
<same stacktrace here>

How to repeat:
Not sure yet.
Looks problem occurs when inventory rediscovery happens (12h interval).
Do not have ability to create debug log on production system .

Suggested fix:
No gaps if graphs.
[5 Nov 2009 21:04] Mark Leith 
This is a duplicate of Bug#47947 - and is a consequence of having a large number of tables causing INFORMATION_SCHEMA queries to hog the stats collection thread. 

Closing as a duplicate, see Bug#47947 for further updates.
[7 Dec 2009 7:36] Andrii Nikitin 
Just a note: I am receiving this error when restoring image (with active MEM setup) in VBox:

2009-12-07 08:07:18: (critical) exception received from server: Internal error: known items are required for list_instances processing
2009-12-07 08:07:29: (critical) network-io.c:841: starting task 6 for mysql::server[38dc335b-0a09-4ef3-84a0-3724fe936ac1]
[8 Jun 2010 10:59] Roel Van de Paar 
Seeing this exact same issue on Agent 2.2.1.1717

2010-06-08 04:19:45: (critical) exception received from server: Internal error: known items are required for list_instances processing

-----

2010-06-08 04:19:45,425  WARN [http-18080-2:com.mysql.agentTask] Client IP: 127.0.0.1
java.lang.RuntimeException: known items are required for list_instances processing
at com.mysql.etools.monitor.bo.Agent.processInventoryTasks(Agent.java:944)
at com.mysql.etools.monitor.bo.Agent.processTasks(Agent.java:578)
...
[16 Aug 2010 11:50] Simon Mudd 
Also seen in 2.2.2.1729

2010-08-16 13:10:49: (critical) network-io.c:312: successfully reconnected to dashboard at https://yyyyyy@xxxxxxxx.prod.xxxx.xxxxxxxx.com:443/heartbeat
2010-08-16 13:11:07: (critical) exception received from server: Internal error: known items are required for list_instances processing
[16 Aug 2010 12:10] Simon Mudd 
checking the agent logs I see this seems to happen periodically:

2010-08-12 01:02:07: (critical) exception received from server: Internal error: known items are required for list_instances processing
2010-08-12 13:03:07: (critical) exception received from server: Internal error: known items are required for list_instances processing
2010-08-13 01:04:08: (critical) exception received from server: Internal error: known items are required for list_instances processing
2010-08-13 13:05:09: (critical) exception received from server: Internal error: known items are required for list_instances processing
2010-08-14 01:06:06: (critical) exception received from server: Internal error: known items are required for list_instances processing
2010-08-14 13:07:07: (critical) exception received from server: Internal error: known items are required for list_instances processing
2010-08-15 01:08:08: (critical) exception received from server: Internal error: known items are required for list_instances processing
2010-08-15 13:09:09: (critical) exception received from server: Internal error: known items are required for list_instances processing
2010-08-16 01:10:06: (critical) exception received from server: Internal error: known items are required for list_instances processing
2010-08-16 13:11:07: (critical) exception received from server: Internal error: known items are required for list_instances processing
[19 Apr 2011 8:37] chatchai komrangded 
How about this bug status ?

I still waitting your to solve this issue as soons as possible.

Regard,
[2 Nov 2011 5:08] Pavel Dobryakov 
Also regularly confronted with the problem. Are there any plans to correct?
[28 Nov 2011 21:38] Joshua Horton 
Also experiencing this problem with mysqlmonitoragent-2.3.7.2103-linux-glibc2.3-x86-64bit-installer.bin operating against 5.5.12-nsa5-log Ubuntu.  It is indeed severe since after about four hours the agents stop sending data and the graphs go blank.  Nothing fancy here, just basic default behavior.  Would be interested in learning that this has been solved.  I'll be listening for updates.
[5 Dec 2011 14:08] Joshua Horton 
I couldn't go 8 hours without the graphs starting to lag behind, and I had to restart the agents because nothing I did helped them catch back up.  After analyzing my issue I determined that it was a case of the agent losing contact with the database, even though the database was completely available.  So I changed the following parameters from their defaults, and I have been running happily for about five days now with no problems.

agent-connect-timeout = 600
agent-IP-version = 4  # we're connecting by socket but wanted to normalize
agent-net-timeout = 600

HTH.

Regards,
Josh
[12 Mar 2013 11:56] Arnaud Adant 
I confirm that the above workaround solved the problem :

agent-connect-timeout = 600
agent-IP-version = 4  
agent-net-timeout = 600