MySQL Bugs: #46052: Agent doesn't report data & pegs CPU when clock is set back while agent running

Bug #46052	Agent doesn't report data & pegs CPU when clock is set back while agent running
Submitted:	8 Jul 2009 18:01	Modified:	2 Jun 2010 8:15
Reporter:	Andy Bang	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Enterprise Monitor: Agent	Severity:	S2 (Serious)
Version:	2.1.0.1074	OS:	Any
Assigned to:	Kay Roepke	CPU Architecture:	Any

Description:
If you set the clock back on the Agent box while the Agent is running, it doesn't report any data to the Service Manager and pegs the CPU at 99%.

How to repeat:
1) Start Service Manager on one machine
2) Start Agent on another, where the time on this second box is in sync with the time on the Service Manager box
3) Let it run for a bit to just see that everything is OK
     o Observe that the agent shows as "up" on the Monitor tab
     o Observe that "Last Agent Contact" in meta info updates when the page is refreshed and is in synch with local time on the Service Manager box
     o Observe that "Reporting Delay Milliseconds" in meta info is reasonable
4) Set the time on the Agent box *back* 15 minutes
5) Wait a minute or two and refresh the Monitor tab
     o Observe that the agent shows as "down" on the Monitor tab
     o Observe that "Last Agent Contact" in meta info does *not* change when the page is refreshed
     o Observe that "Reporting Delay Milliseconds" in meta info stays the same as it was before
     o Observe that there is a "MySQL Agent Not Reachable" critical alert
     o Observe that CPU on the agent box is pegged at 99%
     o Observe that this continues for about 15 minutes
6) Wait a little longer than 15 minutes
     o Observe that the agent shows as "up" on the Monitor tab
     o Observe that the mysqld it's monitoring shows as "down"
     o Observe that "Last Agent Contact" in meta info updates when the page is refreshed and is now in synch with local time on the Service Manager box
     o Observe that "Reporting Delay Milliseconds" in meta info is about equivalent to 15 minutes
     o Observe that CPU on the agent box is no longer pegged at 99%
7) Wait a few more minutes and refresh the Monitor tab
     o Observe that the mysqld the agent is monitoring shows as "up" now

Suggested fix:
1) Detect time changing significantly and rectify the schedules.
2) Don't peg the CPU.

Jan Kneschke writes: 
comments added the review board. Minor issues which can be corrected without another review round. Approved.

Keith Russell writes: 
Patch installed in version => 2.2.0.1536.

Keith Russell writes: 
Patch installed in versions => 2.1.1.1108.

Diego Medina writes: 
Verified fixed on 2.1.1.1108

A note has been added to the 2.1.1 and 2.2.0 changelogs: 

        When using &merlin_agent;, if the current configured time went                                                                                     
        backwards (for example during time correction), then the                                                                                           
        &merlin_agent; would stop reporting data and induce a high                                                                                         
        load on the machine running &merlin_agent;