Bug #43239 agent does not update timestamps when data collections disappear
Submitted: 26 Feb 2009 18:46 Modified: 4 Mar 2009 10:36
Reporter: Darren Oldag
Status: Closed
Category:Monitoring: Agent Severity:S2 (Serious)
Version:2.0, 2.1 OS:Any
Assigned to: MC Brown Target Version:

[26 Feb 2009 18:46] Darren Oldag
Description:
When a data collection becomes invalid in the course of a running agent, the agent sends
NULLs for those collection values.  However, the timestamps that it sends with the values
are the same timestamps from the last-good-value that was collected.

Because of key-constraints on the service manager side, MEM drops on the floor anything
sent with duplicate timestamps, thus the NULLs from the agent are never processed.  Some
things (like replication discovery) depend on the NULLs to know when something has become
invalid.

How to repeat:
monitor a slave with a 2.0/2.1 agent with debug logging
change master to master_host='' on slave.
observe the following in the logs:

      <attribute>Master_ip</attribute>
     </target>
     <utc>2009-02-26T17:13:02.114Z</utc>
     <value>10.20.30.20</value>
    </datum>

      <attribute>Master_ip</attribute>
     </target>
     <utc>2009-02-26T17:13:02.114Z</utc>
    </datum>

the timestamps are the same, but the value went from 10.20.30.20 to NULL

Suggested fix:
update the timestamps when the NULLs are collected
[27 Feb 2009 21:57] Jan Kneschke
Pushed into trunk and 2.0:

revno: 1212
revision-id: jan@mysql.com-20090227205320-85e9errvb1gcbbdr
parent: merlin@dl380-g5-a.mysql.com-20090225212619-q0gnuiljg3pev7pt
committer: jan@mysql.com
branch nick: mysql-lb-2.0
timestamp: Fri 2009-02-27 21:53:20 +0100
message:
  merged fixed for #43239 over

revno: 1204
revision-id: jan@mysql.com-20090227205140-dj6ozcrbgz1m9od5
parent: kay@sun.com-20090225171347-7tw0ncdzxj6efgpa
committer: jan@mysql.com
branch nick: mysql-lb
timestamp: Fri 2009-02-27 21:51:40 +0100
message:
  update ->values_since after the removed all values (fixes #43239)
[3 Mar 2009 20:47] Diego Medina
Verified fixed on 2.0.5.7148

The logs now show:

      <classname>slavestatus</classname>
      <instance>12515cdc-8c00-4223-9d2a-2666a403512c</instance>
      <attribute>Master_ip</attribute>
     </target>
     <utc>2009-03-03T19:43:00.488Z</utc>
     <value>192.168.2.1</value>

--
      <classname>slavestatus</classname>
      <instance>12515cdc-8c00-4223-9d2a-2666a403512c</instance>
      <attribute>Master_ip</attribute>
     </target>
     <utc>2009-03-03T19:44:00.500Z</utc>
    </datum>
[4 Mar 2009 10:36] Tony Bedford
An entry was added to the 2.0.5 changelog:

When a data collection became invalid, the agent sent NULLs for those collection values.
However, the timestamps that it sent with the values were the timestamps from the last
valid value that was collected.

Due to key constraints on the Service Manager side, MySQL Enterprise Monitor disregarded
anything sent with duplicate timestamps, thus the NULLs received from the agent were
never processed. Some mechanisms, such as replication discovery, depend on the NULLs to
identify a situation where data has become invalid.