Bug #37816 max_connections event is blocking with an open "success" alert
Submitted: 2 Jul 2008 18:29 Modified: 11 Nov 2008 13:52
Reporter: Matthew Lord Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Enterprise Monitor: Advisors/Rules Severity:S2 (Serious)
Version:1.3.2.9118 OS:Any
Assigned to: Andy Bang CPU Architecture:Any
Tags: advisor, alert, max_connections, mem, rule

[2 Jul 2008 18:29] Matthew Lord
Description:
The "Maximum Connection Limit Nearing Or Reached" advisor does not generate a new 
(Critical Alert) event when there is an open info (success) event.

This does not appear to be a general issue but rather seems to be specific to
this advisor.

How to repeat:
Install 1.3.2.9118 Monitor and Agent

Schedule the "Maximum Connection Limit Nearing Or Reached" to run every minute

Set max_connections to 10

Wait for an open info (success) event to be generated

Open 10 connections to mysqld

....  notice no new Alerts for this event

Close the open info (success) event.

Now you should see a new critical alert event generated.

Suggested fix:
obvious
[2 Jul 2008 20:42] Sloan Childers
- start from a clean MEM repository
- configure application
- set global max_connections=10; on the monitored database
- schedule Maximum Connection Limit Nearing or Reached rule
- filter on all severity and all status and refresh events page until we have a success event for this rule
- add connections by starting mysql clients until Threads_connected=10
- wait a few minutes
- refresh events page over and over while waiting impatiently
- rule fires properly and events page shows the critical alarm

Either I don't have the steps down on the "reproduce" or maybe the events page needed a refresh and we have a timing issue that made us look broken?
[3 Jul 2008 10:27] Mark Leith
Here's the listed evaluated expression:
(6592 > 10800) && (((11 / 10) * 100) > 75)

Is 6592 greater than 10800? :)

This is a 'warm up period' keyed off of the Uptime status variable included within the rule, which is actually stopping this from firing. 

I'm a little doubtful on why this warm up period is on this rule (it's not buffer related, which is what we usually add this kind of thing to, allowing the buffers to 'warm' before we generate alerts on bad hit ratios etc.). 

Andy - any comments on that? Did we add it for some reason I can't remember now?
[3 Jul 2008 18:05] Sloan Childers
I talked with Leith and Andy and it looks like we do not need a warmup period on this rule.  The suggested fix is to edit the rule and remove the warmup period.

I'm assigning this bug to our rule czar Andy to make a fix before our next rules update.
[7 Jul 2008 19:21] Gary Whizin
Problem is that rule expression includes 3-hour "warm up" period. Workaround is to wait for warm up period or to remove up-time check from the rule expression. Will fix in next Advisor release
[29 Jul 2008 21:26] Andy Bang
In 1.3: Committed revision 9183.

In 2.0: Pushed up to revision 213.
[5 Aug 2008 19:59] Bill Weber
re-opening since the latest advisor build for 1.3 (1.3.0.9189) still has Uptime in the Expression for the Maximum Connection Limit Nearing Or Reached rule - note that it is removed in the latest advisor build for 2.0 (2.0.0.7020)
[10 Oct 2008 17:38] Andy Bang
It was fixed in 9189, but I hadn't changed the version number of the rule so if you just did a rule update instead of a fresh install, you wouldn't see it.  I've fixed that now.
[10 Oct 2008 18:09] Keith Russell
Patch applied in versions => 1.3.0.9210
[23 Oct 2008 0:17] Bill Weber
fixed in Advisor bundles 1.3.0.9217 and 2.0.0.7083
[11 Nov 2008 13:52] Tony Bedford
An entry was added to the 1.3 and 2.0 changelogs:

The “Maximum Connection Limit Nearing Or Reached” advisor did not generate a new Critical Alert event when there was an open info success event.