Bug #55680 MySQL Connector/J memory leak
Submitted: 2 Aug 2010 10:06 Modified: 14 May 2014 19:03
Reporter: Ivan Prisyazhniy Email Updates:
Status: Closed Impact on me:
None 
Category:Connector / J Severity:S2 (Serious)
Version:5.1.13 OS:Linux (x64)
Assigned to: Alexander Soklakov CPU Architecture:Any
Tags: memoryleak

[2 Aug 2010 10:06] Ivan Prisyazhniy
Description:
We are using Hibernate 3.5.3-final, c3p0-0.9.1 and mysql-connector-java-5.1.13 in our project. When the server looses connection to database, our special thread is going to test if database is available. It makes it every certain timeout while database is not available. We have our JDBC connection string with profileSQL enabled and custom logger configured.

So, we found that there is factory - ProfilerEventHandlerFactory. It has a static final field CONNECTIONS_TO_SINKS. It seems that all dead connections are accumulated there. Driver have eaten all available memory after 67 hours of working on trying to restore db connections. Then OutOfMemory exception was caught and head dump made. Max heap size was 250 mb. Drive consumed more than 100mb.

The problem is that there in the connector driver, a lot of instances  of JDBC4Connection were created (about 10663) and garbage collector was unable to collect dead ones. Lot of memory is consumed there by ConnectionPropertiesImpl classes. Dead connections gives a memory leak if you try to check out a new one.

We didn't test configuration with profileSQL disabled.

How to repeat:
1. Setup standard project with Hibernate 3.5.3-final, c3p0-0.9.1 and mysql-connector-java-5.1.13. Set maximum heap size to 100mb for example.
2. Initialize normal hibernate session in your test method. (new db connection should be created)
3. After that switch off the database. And try to check out new connections to db in the eternal loop.

we use following connection string:
jdbc:mysql://server:3306/db?characterEncoding=UTF-8&rewriteBatchedStatements=false&profileSQL=true&enableQueryTimeouts=false&cachePrepStmts=true&cacheCallableStatements=true&cacheServerConfiguration=true&useLocalSessionState=true&elideSetAutoCommits=true&alwaysSendSetIsolation=false

Suggested fix:
When driver caught fatal sql exception that the link is dead, it should remove it from that profiler sink queue. But may be problem is not that. It's guessing.
[2 Aug 2010 10:17] Tonci Grgin
Hi Ivan and thanks for your report.

If you think there is a leak in c/J you should attach standalone test case proving the point and not involve c3p0 or Hibernate.

Can you repeat the leak without 3rd party SW?
[23 Aug 2010 7:32] Tonci Grgin
So? Also, "We didn't test configuration with profileSQL disabled.", profileSQL is not meant for production environments (in fact, no profiling should run on production...).
[24 Aug 2010 14:32] Ivan Prisyazhniy
Testing memory leak when a lot of new dead connections created with profileSQL=true. Shows size of CONNECTIONS_TO_SINKS afterOOM

Attachment: mysql-noconn-test.tar.gz (application/x-gzip, text), 9.36 KiB.

[24 Aug 2010 14:40] Ivan Prisyazhniy
To run mysql-noconn-test you should have free 3333 port and put mysql-connector-java-5.1.13-bin.jar to lib folder. 

To run the test you have to use -Xms, -Xmx flags for vm like following:
java -Xms256M -Xmx256M -jar mysql-noconn-test.jar

On my machine, heap will be filled after ~6287 connection.
[24 Aug 2010 14:48] Ivan Prisyazhniy
Also, we don't use connector in production with profileSQL enabled. 
This bug is not a big problem for us in dev, i just thought you will be interested in the report.
[25 Aug 2010 8:09] Tonci Grgin
Ivan, of course we are interested! Thank you very much for your contribution.
It is just that we do have limited resources so I have to prioritize thus my primary goal was to asses the potential damage your problem might cause in real life.

Thanks again for your interest in MySQL and time put into this.
[29 Apr 2013 9:53] Alexander Soklakov
Verified with c/J 5.1.25.
[14 May 2014 19:03] Daniel So
Added the following entry into the Connector/J 5.1.31 changelog:

"If profileSQL was enabled, a memory leak would occur after a connection was lost and continuous attempts were made to reconnect. It was because ProfilerEventHandlerFactory kept a map in which dead connections kept on accumulating. This fix refactors the ProfilerEventHandlerFactory and eliminates that map."