MySQL Bugs: #45924: monitor log gets flooded with error messages when retrying on db errors

Bug #45924	monitor log gets flooded with error messages when retrying on db errors
Submitted:	2 Jul 2009 22:03	Modified:	21 Jul 2009 15:52
Reporter:	Marcos Palacios	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Enterprise Monitor: Server	Severity:	S3 (Non-critical)
Version:	2.1.0.1073	OS:	Any
Assigned to:	Mark Matthews	CPU Architecture:	Any

Description:
The monitor log gets flooded with ERROR/WARN messages when the service manager is retrying for a mysqld connection and finally issuing a cry for help; the messages are like these shown below (every second):

2009-07-02 16:52:23,440 WARN [http-18080-4:com.mysql.sql] java.lang.Exception: MySQL server not running or accepting connections, retrying 30 times or 46 seconds, whichever expires first. Exception was: Communications link failure
Last packet sent to the server was 0 ms ago.
2009-07-02 16:52:24,455 WARN [Purger:com.mysql.sql] java.lang.Exception: MySQL server not running or accepting connections, retrying 46 times or 151 seconds, whichever expires first. Exception was: Communications link failure
Last packet sent to the server was 0 ms ago.
2009-07-02 16:52:25,471 WARN [http-18080-5:com.mysql.sql] java.lang.Exception: MySQL server not running or accepting connections, retrying 28 times or 34 seconds, whichever expires first. Exception was: Communications link failure
Last packet sent to the server was 0 ms ago.
2009-07-02 16:52:26,377 WARN [http-18080-9:com.mysql.sql] java.lang.Exception: MySQL server not running or accepting connections, retrying 33 times or 64 seconds, whichever expires first. Exception was: Communications link failure
Last packet sent to the server was 0 ms ago.

Comment from dev: "no need to log mysqld is down 50 times for each thread in the connection pool; i think logs will fill and if we have enough agents we might croak on the restart of mysqld".

How to repeat:
- Kill the mysqld internal repository while the application is running and agents are connected
- Check the monitor log for those messages

Gary Whizin writes: 
Wondering why log4j code to remove duplicate log messages not catching these?

Mark Matthews writes: 
Note that you'll still get "n" messages, one for each failed attempt by the first thread that found the failure. The "guard" resets after the attempt ultimately fails or succeeds.

Keith Russell writes: 
Patch applied in versions => 2.1.0.1074.

Marcos Palacios writes: 
Verified fixed in build 2.1.0.1074.

An entry was added to the 2.1.0 changelog:

The Enterprise Monitor log was flooded with ERROR/WARN messages when the Service Manager was retrying for a mysqld connection:

2009-07-02 16:52:23,440  WARN [http-18080-4:com.mysql.sql] java.lang.Exception: MySQL
server not running or accepting connections, retrying  30 times or 46 seconds, whichever
expires first. Exception was: Communications link failure
Last packet sent to the server was 0 ms ago.
2009-07-02 16:52:24,455  WARN [Purger:com.mysql.sql] java.lang.Exception: MySQL server not
running or accepting connections, retrying  46 times or 151 seconds, whichever expires
first. Exception was: Communications link failure
Last packet sent to the server was 0 ms ago.
2009-07-02 16:52:25,471  WARN [http-18080-5:com.mysql.sql] java.lang.Exception: MySQL
server not running or accepting connections, retrying  28 times or 34 seconds, whichever
expires first. Exception was: Communications link failure
Last packet sent to the server was 0 ms ago.
2009-07-02 16:52:26,377  WARN [http-18080-9:com.mysql.sql] java.lang.Exception: MySQL
server not running or accepting connections, retrying  33 times or 64 seconds, whichever
expires first. Exception was: Communications link failure
Last packet sent to the server was 0 ms ago.