MySQL Bugs: #55516: catalina.out gows too fast for many agents when repository is down

Bug #55516	catalina.out gows too fast for many agents when repository is down
Submitted:	23 Jul 2010 15:51	Modified:	9 Sep 2010 14:20
Reporter:	Andrii Nikitin	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Enterprise Monitor: Server	Severity:	S3 (Non-critical)
Version:	2.2.2	OS:	Any
Assigned to:	Andy Bang	CPU Architecture:	Any

Description:
When MEM repository is not reachable, messages like below appear in catalina.out :

Jul 23, 2010 11:29:47 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet heartbeat threw exception
org.hibernate.exception.GenericJDBCException: Cannot open connection
org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126)
org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114)
org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
...

Size of single message is 4Kb, and it is printed for each agent 3 times per minute.

This means that with 400 agents running, log file will grow 5Mb per minute or almost 300Mb per hour.

It real environment it did grow almost 1G at night just in few hours, which can cause problems

How to repeat:
Start monitoring with single agent, stop MEM repository, see that ~30 error messages appeared in catalina.out in 10 minutes and it grew on 120Kb

Start monitoring with three agents, stop MEM repository, see that ~90 error messages appeared in catalina.out in 10 minutes and it grew on 360Kb

Start monitoring with 400 agents, stop MEM repository, see that ~1200 error messages appeared in catalina.out in 10 minutes and it grew on 48Mb

Suggested fix:
Do not grow logs so fast.

Attachment: 10425_tomcat-juli-adapters.jar (application/java-archive, text), 20.44 KiB.



Attachment: 10426_tomcat-juli.jar (application/java-archive, text), 54.42 KiB.



Attachment: 10427_log4j.properties (application/octet-stream, text), 1.31 KiB.

Bill Weber writes: 
verified fixed in build 2.2.3.1737

Setting to documenting for changelogs.

An entry has been added to the 2.2.3 changelog: 

        When the MySQL server supporting &merlin_server; is                                                                              
        unavailable, the <filename>catalina.out</filename> file is                                                                       
        populated with error messages for each &merlin_agent;                                                                            
        communicating with &merlin_server;. The quantity and frequency                                                                   
        of the messages can cause the error log to grow at a very fast                                                                   
        rate. The logging configuration has been altered so that                                                                         
        Tomcat-specific errors go to <literal>tomcat.log</literal>,                                                                      
        rotating every 10MB for up to 10 logs. Application errors are                                                                    
        reported in <filename>mysql-monitor.log</filename>.