Bug #47165 Infobright and MEM
Submitted: 7 Sep 2009 10:01 Modified: 24 May 2010 12:39
Reporter: Bogdan Kecman Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Enterprise Monitor Severity:S4 (Feature request)
Version:2.x OS:Any
Assigned to: Michael Schuster CPU Architecture:Any

[7 Sep 2009 10:01] Bogdan Kecman
Description:
MEM has issues with MySQL server using Infobright storage engine.

How to repeat:
.

Suggested fix:
.
[10 Sep 2009 4:34] MySQL Verification Team
More info from same customer, his log files show agent is getting DuplicateAgentUuidException, yet no entry in MEM inventory_instances table.

I tested with agent debug on infobright. brighthouse is default engine, have_innodb=no. So agent starts up tries to create mysql.inventory, fails due to following error:

2009-06-21 01:36:19: (critical) agent_mysqld.c:284: mysql_real_query('CREATE TABLE mysql.inventory ( name VARCHAR(64) NOT NULL PRIMARY KEY,
 value VARCHAR(64))') on 'mysql' failed: Too many key parts specified; max 0 parts allowed (mysql-errno = 1070)
2009-06-21 01:36:19: (critical) agent_mysqld.c:284: mysql_real_query('SELECT value FROM mysql.inventory WHERE name = 'hostid'') on 'mysql' 
failed: Table 'mysql.inventory' doesn't exist (mysql-errno = 1146)
2009-06-21 01:36:19: (warning) job_collect_mysql.c:781: mysql has no UUID set

Fix is to change default engine temporarily to myisam. That allows mysql.inventory to be created and populated. Agent starts getting MySQL info,status's,etc but gets signal 11 after show innodb status:

2009-06-18 22:11:56: (message) mysql-query (mysql): SELECT COUNT(*) AS user_count FROM mysql.user WHERE user=''
2009-06-18 22:11:56: (message) mysql-query (mysql): SHOW /*!50000 ENGINE */ INNODB STATUS
2009-06-18 22:11:56: (debug) chassis.c:282: 5065 returned: 5065
2009-06-18 22:11:56: (message) chassis.c:304: [angel] PID=5065 died on signal=11 (it used 0 kBytes max) ... waiting 3min before restart
2009-06-18 22:11:58: (debug) chassis.c:244: we are the child: 5085
2009-06-18 22:11:58: (message) mysql-proxy 0.7.0 started
2009-06-18 22:11:58: (debug) chassis.c:1091: current RLIMIT_NOFILE = 4398046512128 (hard: 577792033385921489)
2009-06-18 22:11:58: (debug) chassis.c:1095: trying to set new RLIMIT_NOFILE = 4398046519296 (hard: 577792033385921489)
2009-06-18 22:11:58: (critical) chassis.c:1097: could not raise RLIMIT_NOFILE to 8192, Invalid argument (22). Current limit still 4294968320.
2009-06-18 22:11:58: (message) MySQL Monitor Agent 2.0.5.7153 started.

... then restarts ...

Then server returns DuplicateAgentUuidException:

2009-06-18 22:11:58: (debug) <-- received: <?xml version='1.0'?><exceptions><error><![CDATA[E1402: DuplicateAgentUuidException: [a53709ac-587f-4d1c-aeb4-98aab4f3f4ba]]]></error></exceptions>
2009-06-18 22:11:58: (message) network-io.c:250: encoding = 0.04 ms, request = 30.25 ms (incl. response 0.32 ms)
2009-06-18 22:11:58: (critical) network-io.c:256: successfully reconnected to dashboard at http://agent:test@127.0.0.1:18080/heartbeat
2009-06-18 22:11:58: (critical) exception received from server: E1402: DuplicateAgentUuidException: [a53709ac-587f-4d1c-aeb4-98aab4f3f4ba]

Checking MEM inventory_instances shows no uuid. 

mysql-monitor-log shows:

com.mysql.etools.exception.DuplicateAgentUuidException: E1402: DuplicateAgentUuidException: [a53709ac-587f-4d1c-aeb4-98aab4f3f4ba]
	at com.mysql.etools.monitor.bo.SessionManager.getSession(SessionManager.java:76)
	at com.mysql.etools.monitor.bo.HeartbeatAcceptor.acceptHeartbeat(HeartbeatAcceptor.java:42)
	at com.mysql.etools.monitor.bo.HeartBeatCommandProcessor.processRequest(HeartBeatCommandProcessor.java:102)
	at com.mysql.merlin.server.MerlinServlet$1.processRequest(MerlinServlet.java:412)
	at com.mysql.merlin.server.agent.HeartbeatServlet$1.processRequest(HeartbeatServlet.java:62)
	at com.mysql.merlin.server.MerlinServlet.doCommon(MerlinServlet.java:365)
	at com.mysql.merlin.server.MerlinServlet.doPost(MerlinServlet.java:504)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.mysql.util.RequestCounterFilter.doFilter(RequestCounterFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
	at java.lang.Thread.run(Unknown Source)

After receiving this, I restarted monitor, same dupuuid error, I restarted agent with same results. I also tried infobright socket and ip connection from fresh mem database and rm'd mysql.inventory each time with the same results.
[10 Sep 2009 4:49] MySQL Verification Team
Forgot to mention just in case i tested MySQL with skip-innodb as well, no problem there. This was using community Infobright install and mem/agent 2.0.5.7153. If need more agent debug out let me know.
[10 Sep 2009 6:30] Simon Mudd
Why are you still checking the innodb status when there is no innodb engine. This is something I reported previously on other instances (MyISAM only, where skip_innodb was set).

I'd have to find the ticket but I suggested checking for the agent each time that SHOW ENGINE INNODB STATUS is run, or if not caching the result and checking less frequently.

Anyway, good you are able to make progress.
[10 Sep 2009 11:09] Mark Leith
It's not as simple as that Simon, especially with the upcoming InnoDB plugin etc. - which can all be installed and start using them at run time.

We must continue to check for it, at the same pace as is configured, in the case that it does become available.
[11 Sep 2009 4:51] MySQL Verification Team
Same behavior on 2.1.1093 which is the version customer has after upgrading, except there is now no error with mysql.inventory creation as MyISAM is now specified as engine. Rest is same as I mention (sig 11, dupuuid,etc).
[11 Sep 2009 15:51] Enterprise Tools JIRA Robot
Diego Medina writes: 
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000008
[Switching to process 79869 thread 0x2903]
0x00362538 in agent_item_class_update_mysqld_from_innodbstatus ()
(gdb) bt
#0  0x00362538 in agent_item_class_update_mysqld_from_innodbstatus ()
#1  0x000fb556 in g_hash_table_foreach ()
#2  0x003628a9 in agent_item_class_update_all_from_innodbstatus ()
#3  0x00359d5a in knownitems_class_iter ()
#4  0x000fb556 in g_hash_table_foreach ()
#5  0x000fb556 in g_hash_table_foreach ()
#6  0x00359ea7 in job_list_knowndataitems_get_items ()
#7  0x00359f7f in job_list_knowndataitems_thread ()
#8  0x001318de in g_thread_create_proxy ()
#9  0x95c05155 in _pthread_start ()
#10 0x95c05012 in thread_start ()

===============
[9 Dec 2009 14:18] Enterprise Tools JIRA Robot
Mark Matthews writes: 
Fixed in trunk, are we going to backport this to 2.1?
[11 Dec 2009 2:40] Roel Van de Paar
Workaround: do not give the agent userid SUPER privileges 

See: http://www.infobright.org/forums/viewthread/344/
[10 May 2010 15:16] Enterprise Tools JIRA Robot
Keith Russell writes: 
Patch installed in versions => 2.1.2.1161.
[17 May 2010 21:00] Enterprise Tools JIRA Robot
Diego Medina writes: 
Verified fixed on 2.1.2.1161  , you can now monitor an infobright instance.
[24 May 2010 12:39] MC Brown
A note has been added to the 2.1.2 changelog: 

    &merlin_agent; would fail to monitor MySQL servers using the InfoBright storage engine.