Bug #67954 ClassLoader leak in ConnectionImpl
Submitted: 19 Dec 2012 21:27 Modified: 22 Jan 2013 23:17
Reporter: Tom Jansen Email Updates:
Status: Closed Impact on me:
Category:Connector / J Severity:S2 (Serious)
Version:5.1.22 OS:Any
Assigned to: Mark Matthews CPU Architecture:Any
Tags: classloader leak stacktrace garbage collection pool

[19 Dec 2012 21:27] Tom Jansen
After a chasing a classloader garbage collection leak, I've found the following problem in our environment where connections are pooled via commons-dbcp.

The mysql connector jar is placed in tomcat7/lib and thus loaded by the StandardClassLoader. The connection pool is loaded in the same classloader.

When a connection is requested via the pool, it is created by the pool and handed to the webapp, which it loaded via its own WebappClassLoader instance.

When the webapp is done with the connection, it hands it back to the pool, but it is not close()'ed because it will be reused. The pool keeps a reference to the Connection. The webapp should be able to undeploy now, and its WebappClassloader garbage collected. This is not happening, and is thus creating  a PermGen memory leak. 

The garbage collection is not happening because of the same reason as described here: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6916498

In short, a stracktrace is created upon creating a ConnectionImpl. This trace holds a reference to Class-es loaded by the WebappClassloader. They don't show up in a heap analysis tool, but they are there and prevent garbage collection of the WebappClassloader. (as described in the oracle/sun bug above)

Offending method:
protected ConnectionImpl(String hostToConnectTo, int portToConnectTo, Properties info, String databaseToConnectTo, String url)

Offending code:
this.pointOfOrigin = new Throwable();

The pointOfOrigin is only used for logging, and null'ing it causes no trouble except for less useful log messages, but not GC'ing the WebappClassloader instance causes 25MB of PermGen space to be leaked upon every deploy of our app.

How to repeat:
- Place mysql jar in tomcat7/lib
- define mysql jdbc database connection in tomcat7/conf/server.xml (default pooled by tomcat/commons-dbcp).
- get connection in webapp via JDNI (and thus via the pool)
- close connection (and thus hand it back to the pool)
- try to unload webapp, force full GC and observe that the WebappClassloader is not being GC'ed

Suggested fix:
Do not keep pointOfOrigin Throwable in ConnectionImpl and lose logging with backtraces in realClose()


create String version of the trace and use that for logging instead.


make filling in pointOfOrigin configurable for those who need the logging and warn about the side-effects in the docs.
[8 Jan 2013 20:59] Mark Matthews
Fixed for .23
[22 Jan 2013 23:17] John Russell
Added to changelog for 5.1.23: 

Stack trace used for point-of-origin in log and exception messages
caused a Permgen leak when an application was redeployed, because the
WebappClassloader could not be garbage collected. We no longer store
the entire stack trace, only the calling class and method, and only
when using the usage advisor or when profiling.