Description:
After a chasing a classloader garbage collection leak, I've found the following problem in our environment where connections are pooled via commons-dbcp.
The mysql connector jar is placed in tomcat7/lib and thus loaded by the StandardClassLoader. The connection pool is loaded in the same classloader.
When a connection is requested via the pool, it is created by the pool and handed to the webapp, which it loaded via its own WebappClassLoader instance.
When the webapp is done with the connection, it hands it back to the pool, but it is not close()'ed because it will be reused. The pool keeps a reference to the Connection. The webapp should be able to undeploy now, and its WebappClassloader garbage collected. This is not happening, and is thus creating a PermGen memory leak.
The garbage collection is not happening because of the same reason as described here: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6916498
In short, a stracktrace is created upon creating a ConnectionImpl. This trace holds a reference to Class-es loaded by the WebappClassloader. They don't show up in a heap analysis tool, but they are there and prevent garbage collection of the WebappClassloader. (as described in the oracle/sun bug above)
Offending method:
protected ConnectionImpl(String hostToConnectTo, int portToConnectTo, Properties info, String databaseToConnectTo, String url)
Offending code:
this.pointOfOrigin = new Throwable();
The pointOfOrigin is only used for logging, and null'ing it causes no trouble except for less useful log messages, but not GC'ing the WebappClassloader instance causes 25MB of PermGen space to be leaked upon every deploy of our app.
How to repeat:
- Place mysql jar in tomcat7/lib
- define mysql jdbc database connection in tomcat7/conf/server.xml (default pooled by tomcat/commons-dbcp).
- get connection in webapp via JDNI (and thus via the pool)
- close connection (and thus hand it back to the pool)
- try to unload webapp, force full GC and observe that the WebappClassloader is not being GC'ed
Suggested fix:
Do not keep pointOfOrigin Throwable in ConnectionImpl and lose logging with backtraces in realClose()
or
create String version of the trace and use that for logging instead.
or
make filling in pointOfOrigin configurable for those who need the logging and warn about the side-effects in the docs.
