MySQL Bugs: #37816: max_connections event is blocking with an open "success" alert

Bug #37816	max_connections event is blocking with an open "success" alert
Submitted:	2 Jul 2008 18:29	Modified:	11 Nov 2008 13:52
Reporter:	Matthew Lord	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Enterprise Monitor: Advisors/Rules	Severity:	S2 (Serious)
Version:	1.3.2.9118	OS:	Any
Assigned to:	Andy Bang	CPU Architecture:	Any
Tags:	advisor, alert, max_connections, mem, rule

Description:
The "Maximum Connection Limit Nearing Or Reached" advisor does not generate a new 
(Critical Alert) event when there is an open info (success) event.

This does not appear to be a general issue but rather seems to be specific to
this advisor.

How to repeat:
Install 1.3.2.9118 Monitor and Agent

Schedule the "Maximum Connection Limit Nearing Or Reached" to run every minute

Set max_connections to 10

Wait for an open info (success) event to be generated

Open 10 connections to mysqld

....  notice no new Alerts for this event

Close the open info (success) event.

Now you should see a new critical alert event generated.

Suggested fix:
obvious

- start from a clean MEM repository
- configure application
- set global max_connections=10; on the monitored database
- schedule Maximum Connection Limit Nearing or Reached rule
- filter on all severity and all status and refresh events page until we have a success event for this rule
- add connections by starting mysql clients until Threads_connected=10
- wait a few minutes
- refresh events page over and over while waiting impatiently
- rule fires properly and events page shows the critical alarm

Either I don't have the steps down on the "reproduce" or maybe the events page needed a refresh and we have a timing issue that made us look broken?

Here's the listed evaluated expression:
(6592 > 10800) && (((11 / 10) * 100) > 75)

Is 6592 greater than 10800? :)

This is a 'warm up period' keyed off of the Uptime status variable included within the rule, which is actually stopping this from firing. 

I'm a little doubtful on why this warm up period is on this rule (it's not buffer related, which is what we usually add this kind of thing to, allowing the buffers to 'warm' before we generate alerts on bad hit ratios etc.). 

Andy - any comments on that? Did we add it for some reason I can't remember now?

I talked with Leith and Andy and it looks like we do not need a warmup period on this rule.  The suggested fix is to edit the rule and remove the warmup period.

I'm assigning this bug to our rule czar Andy to make a fix before our next rules update.

Problem is that rule expression includes 3-hour "warm up" period. Workaround is to wait for warm up period or to remove up-time check from the rule expression. Will fix in next Advisor release

In 1.3: Committed revision 9183.

In 2.0: Pushed up to revision 213.

re-opening since the latest advisor build for 1.3 (1.3.0.9189) still has Uptime in the Expression for the Maximum Connection Limit Nearing Or Reached rule - note that it is removed in the latest advisor build for 2.0 (2.0.0.7020)

It was fixed in 9189, but I hadn't changed the version number of the rule so if you just did a rule update instead of a fresh install, you wouldn't see it.  I've fixed that now.

Patch applied in versions => 1.3.0.9210

fixed in Advisor bundles 1.3.0.9217 and 2.0.0.7083

An entry was added to the 1.3 and 2.0 changelogs:

The “Maximum Connection Limit Nearing Or Reached” advisor did not generate a new Critical Alert event when there was an open info success event.