Bug #43239 agent does not update timestamps when data collections disappear
Submitted: 26 Feb 2009 17:46 Modified: 4 Mar 2009 9:36
Reporter: Darren Oldag Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Enterprise Monitor: Agent Severity:S2 (Serious)
Version:2.0, 2.1 OS:Any
Assigned to: MC Brown CPU Architecture:Any

[26 Feb 2009 17:46] Darren Oldag
Description:
When a data collection becomes invalid in the course of a running agent, the agent sends NULLs for those collection values.  However, the timestamps that it sends with the values are the same timestamps from the last-good-value that was collected.

Because of key-constraints on the service manager side, MEM drops on the floor anything sent with duplicate timestamps, thus the NULLs from the agent are never processed.  Some things (like replication discovery) depend on the NULLs to know when something has become invalid.

How to repeat:
monitor a slave with a 2.0/2.1 agent with debug logging
change master to master_host='' on slave.
observe the following in the logs:

      <attribute>Master_ip</attribute>
     </target>
     <utc>2009-02-26T17:13:02.114Z</utc>
     <value>10.20.30.20</value>
    </datum>

      <attribute>Master_ip</attribute>
     </target>
     <utc>2009-02-26T17:13:02.114Z</utc>
    </datum>

the timestamps are the same, but the value went from 10.20.30.20 to NULL

Suggested fix:
update the timestamps when the NULLs are collected
[27 Feb 2009 20:57] Jan Kneschke
Pushed into trunk and 2.0:

revno: 1212
revision-id: jan@mysql.com-20090227205320-85e9errvb1gcbbdr
parent: merlin@dl380-g5-a.mysql.com-20090225212619-q0gnuiljg3pev7pt
committer: jan@mysql.com
branch nick: mysql-lb-2.0
timestamp: Fri 2009-02-27 21:53:20 +0100
message:
  merged fixed for #43239 over

revno: 1204
revision-id: jan@mysql.com-20090227205140-dj6ozcrbgz1m9od5
parent: kay@sun.com-20090225171347-7tw0ncdzxj6efgpa
committer: jan@mysql.com
branch nick: mysql-lb
timestamp: Fri 2009-02-27 21:51:40 +0100
message:
  update ->values_since after the removed all values (fixes #43239)
[3 Mar 2009 19:47] Diego Medina
Verified fixed on 2.0.5.7148

The logs now show:

      <classname>slavestatus</classname>
      <instance>12515cdc-8c00-4223-9d2a-2666a403512c</instance>
      <attribute>Master_ip</attribute>
     </target>
     <utc>2009-03-03T19:43:00.488Z</utc>
     <value>192.168.2.1</value>

--
      <classname>slavestatus</classname>
      <instance>12515cdc-8c00-4223-9d2a-2666a403512c</instance>
      <attribute>Master_ip</attribute>
     </target>
     <utc>2009-03-03T19:44:00.500Z</utc>
    </datum>
[4 Mar 2009 9:36] Tony Bedford
An entry was added to the 2.0.5 changelog:

When a data collection became invalid, the agent sent NULLs for those collection values. However, the timestamps that it sent with the values were the timestamps from the last valid value that was collected.

Due to key constraints on the Service Manager side, MySQL Enterprise Monitor disregarded anything sent with duplicate timestamps, thus the NULLs received from the agent were never processed. Some mechanisms, such as replication discovery, depend on the NULLs to identify a situation where data has become invalid.