Bug #45016 agent tries to get quan-config before mem-server has set it up
Submitted: 21 May 2009 20:28 Modified: 21 Jul 2009 10:58
Reporter: Marcos Palacios Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Enterprise Monitor: Agent Severity:S3 (Non-critical)
Version:2.1.0.1048 OS:Any
Assigned to: Jan Kneschke CPU Architecture:Any

[21 May 2009 20:28] Marcos Palacios
Description:
On a new mysql-server, the agent gives this critical error (at 2009-05-21 15:59:52 2nd one) in the log:

2009-05-21 15:59:42: (critical) network-io.c:255: successfully reconnected to dashboard at http://agent:merlin@127.0.0.1:18080/heartbeat
2009-05-21 15:59:47: (critical) agent_mysqld.c:641: successfully connected to database at 127.0.0.1:13306 as user marcos (with password: YES)
2009-05-21 15:59:52: (critical) network-io.c:807: starting task 2 for mysql::server[8b1294a5-b40b-457c-b8d3-a834848dc1df]

2009-05-21 15:59:52: (critical) job_collect_lua.c:323: ...ql/enterprise/agent/share/mysql-proxy/items/quan.lua:505: GETing quan-config from http://agent:merlin@127.0.0.1:18080/v2/rest/instance/mysql/StatementAnalysisSupport/8b1294... failed with: 404
8b1294a5-b40b-457c-b8d3-a834848dc1df of type mysql.StatementAnalysisSupport not found

2009-05-21 16:00:00: (critical) (share/mysql-proxy/items/quan.lua:528) setting .analyze_queries for 8b1294a5-b40b-457c-b8d3-a834848dc1df to true
2009-05-21 16:00:00: (critical) (share/mysql-proxy/items/quan.lua:532) setting .auto_explain for 8b1294a5-b40b-457c-b8d3-a834848dc1df to true
2009-05-21 16:00:00: (critical) (share/mysql-proxy/items/quan.lua:536) setting .analyze_worst_queries for 8b1294a5-b40b-457c-b8d3-a834848dc1df to true
2009-05-21 16:00:00: (critical) (share/mysql-proxy/items/quan.lua:540) setting .auto_explain_min_exec_time_us for 8b1294a5-b40b-457c-b8d3-a834848dc1df to 500000

How to repeat:
- install svc mgr
- install agent
- check the agent log
[27 May 2009 11:06] Jan Kneschke
The problem is that we start the internal task to get quan-config in parallel as we send it to the mem-server.

This leads to a race between the mysql::server instances being reported in the heartbeat and the mysql::quanconfig task getting the QUAN config for that server in a second thread. In case the mysql::quanconfig task is faster, you get a 404 if the mysql::server is new.

The fix is to delay the internal task scheduling until we get the confirmation from the mem-server.
[25 Jun 2009 11:29] Jan Kneschke
------------------------------------------------------------
revno: 1398
committer: jan@mysql.com
branch nick: trunk
timestamp: Thu 2009-06-25 12:46:02 +0200
message:
  only process the data for the internal tasks after the we sent it to the mem-server (fixes #45016)
  
    * moved the handling of the data into its own function
[8 Jul 2009 20:45] Enterprise Tools JIRA Robot
Keith Russell writes: 
Patch installed in versions => 2.1.0.1074.
[8 Jul 2009 21:40] Enterprise Tools JIRA Robot
Marcos Palacios writes: 
Verified fixed in agent build 2.1.0.1074.
[21 Jul 2009 10:58] Tony Bedford
An entry was added to the 2.1.0 changelog:

The Agent attempted to obtain the Query Analyzer configuration before the Monitor server has created it. This resulted in the Agent generating the following critical error in the log:

2009-05-21 15:59:42: (critical) network-io.c:255: successfully reconnected to dashboard at
http://agent:merlin@127.0.0.1:18080/heartbeat
2009-05-21 15:59:47: (critical) agent_mysqld.c:641: successfully connected to database at
127.0.0.1:13306 as user marcos (with password: YES)
2009-05-21 15:59:52: (critical) network-io.c:807: starting task 2 for
mysql::server[8b1294a5-b40b-457c-b8d3-a834848dc1df]

2009-05-21 15:59:52: (critical) job_collect_lua.c:323:
...ql/enterprise/agent/share/mysql-proxy/items/quan.lua:505: GETing quan-config from
http://agent:merlin@127.0.0.1:18080/v2/rest/instance/mysql/StatementAnalysisSupport/8b1294
a5-b40b-457c-b8d3-a834848dc1df failed with: 404
8b1294a5-b40b-457c-b8d3-a834848dc1df of type mysql.StatementAnalysisSupport not found

2009-05-21 16:00:00: (critical) (share/mysql-proxy/items/quan.lua:528) setting
.analyze_queries for 8b1294a5-b40b-457c-b8d3-a834848dc1df to true
2009-05-21 16:00:00: (critical) (share/mysql-proxy/items/quan.lua:532) setting
.auto_explain for 8b1294a5-b40b-457c-b8d3-a834848dc1df to true
2009-05-21 16:00:00: (critical) (share/mysql-proxy/items/quan.lua:536) setting
.analyze_worst_queries for 8b1294a5-b40b-457c-b8d3-a834848dc1df to true
2009-05-21 16:00:00: (critical) (share/mysql-proxy/items/quan.lua:540) setting
.auto_explain_min_exec_time_us for 8b1294a5-b40b-457c-b8d3-a834848dc1df to 500000