MySQL Bugs: #37816: max_connections event is blocking with an open "success" alert

Bug #37816	max_connections event is blocking with an open "success" alert
Submitted:	2 Jul 2008 20:29	Modified:	11 Nov 2008 14:52
Reporter:	Matthew Lord
Status:	Closed
Category:	Monitoring: Advisors/Rules	Severity:	S2 (Serious)
Version:	1.3.2.9118	OS:	Any
Assigned to:	Andy Bang	Target Version:	1.3 next advisor update
Tags:	max_connections, alert, rule, advisor, mem
Triage:	D2 (Serious) / R2 (Low) / E2 (Low)

Description:
The "Maximum Connection Limit Nearing Or Reached" advisor does not generate a new 
(Critical Alert) event when there is an open info (success) event.

This does not appear to be a general issue but rather seems to be specific to
this advisor.

How to repeat:
Install 1.3.2.9118 Monitor and Agent

Schedule the "Maximum Connection Limit Nearing Or Reached" to run every minute

Set max_connections to 10

Wait for an open info (success) event to be generated

Open 10 connections to mysqld

....  notice no new Alerts for this event

Close the open info (success) event.

Now you should see a new critical alert event generated.

Suggested fix:
obvious

- start from a clean MEM repository
- configure application
- set global max_connections=10; on the monitored database
- schedule Maximum Connection Limit Nearing or Reached rule
- filter on all severity and all status and refresh events page until we have a success
event for this rule
- add connections by starting mysql clients until Threads_connected=10
- wait a few minutes
- refresh events page over and over while waiting impatiently
- rule fires properly and events page shows the critical alarm

Either I don't have the steps down on the "reproduce" or maybe the events page needed a
refresh and we have a timing issue that made us look broken?

Here's the listed evaluated expression:
(6592 > 10800) && (((11 / 10) * 100) > 75)

Is 6592 greater than 10800? :)

This is a 'warm up period' keyed off of the Uptime status variable included within the
rule, which is actually stopping this from firing. 

I'm a little doubtful on why this warm up period is on this rule (it's not buffer
related, which is what we usually add this kind of thing to, allowing the buffers to
'warm' before we generate alerts on bad hit ratios etc.). 

Andy - any comments on that? Did we add it for some reason I can't remember now?

I talked with Leith and Andy and it looks like we do not need a warmup period on this
rule.  The suggested fix is to edit the rule and remove the warmup period.

I'm assigning this bug to our rule czar Andy to make a fix before our next rules update.

Problem is that rule expression includes 3-hour "warm up" period. Workaround is to wait
for warm up period or to remove up-time check from the rule expression. Will fix in next
Advisor release

In 1.3: Committed revision 9183.

In 2.0: Pushed up to revision 213.

re-opening since the latest advisor build for 1.3 (1.3.0.9189) still has Uptime in the
Expression for the Maximum Connection Limit Nearing Or Reached rule - note that it is
removed in the latest advisor build for 2.0 (2.0.0.7020)

It was fixed in 9189, but I hadn't changed the version number of the rule so if you just
did a rule update instead of a fresh install, you wouldn't see it.  I've fixed that now.

Patch applied in versions => 1.3.0.9210

fixed in Advisor bundles 1.3.0.9217 and 2.0.0.7083

An entry was added to the 1.3 and 2.0 changelogs:

The “Maximum Connection Limit Nearing Or Reached” advisor did not generate a new
Critical Alert event when there was an open info success event.