Bug #25073 rewriteBatchedStatements memory leak on executeBatch
Submitted: 14 Dec 2006 14:40 Modified: 22 Feb 2007 14:18
Reporter: Jason Winnebeck Email Updates:
Status: Closed Impact on me:
None 
Category:Connector / J Severity:S2 (Serious)
Version:3.1.13, 5.0.3, 5.0.4 OS:Any (all)
Assigned to: CPU Architecture:Any
Tags: memory leak

[14 Dec 2006 14:40] Jason Winnebeck
Description:
When using parameters "?rewriteBatchedStatements=true&useServerPrepStmts=false", memory leak occurs when calling PreparedStatement.executeBatch()

Analysis: When calling executeBatch, code flows to executeBatchedInserts, which creates a new PreparedStatement in PreparedStatement.executeBatchedInserts into variable batchedStatement. This statement is never closed and all references to it are lost when method ends. Apparently when a statement is created it registers itself into some map and this map holds a reference to the statement forever. Because of the client-side processing the PS object is very large, causing aggressive memory leak.

How to repeat:
Occurs always when enabling and actually using multi-insert statement rewriting

Suggested fix:
Close the temporary PrepStmt created

patch against 5.0.3: (fix for 5.0.2 and 3.1.13 same, but different lines)

diff -Naur src/com/mysql/jdbc/PreparedStatement.java src-rit/com/mysql/jdbc/PreparedStatement.java
--- src/com/mysql/jdbc/PreparedStatement.java   2006-12-14 09:33:00.000000000 -0500
+++ src-rit/com/mysql/jdbc/PreparedStatement.java       2006-12-14 09:33:38.000000000 -0500
@@ -953,6 +953,8 @@
                        getBatchedGeneratedKeys(batchedStatement);
                }

+    batchedStatement.close();
+
                int[] updateCounts = new int[this.batchedArgs.size()];

                for (int i = 0; i < this.batchedArgs.size(); i++) {
[14 Dec 2006 14:41] Jason Winnebeck
Patch against 5.0.3 source

Attachment: patch (application/octet-stream, text), 489 bytes.

[14 Dec 2006 14:45] Jason Winnebeck
I meant to mark this as S2.
[14 Dec 2006 14:47] Jason Winnebeck
Sorry, patch is against 5.0.4 source and not 5.0.3 but the fix is the same only shifted a few lines.
[14 Dec 2006 18:19] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/16976
[22 Feb 2007 14:05] MC Brown
A note has been added to the 5.0.5 changelog.
[22 Feb 2007 14:18] Jason Winnebeck
Will this be put into the 3.1 series? The 5 series driver may be more functional but the 3.1 series is much faster in our application and the primary driver for our application is performance.
[4 Apr 2007 18:07] Joseph Dunleavy
This is a HPROF output when I have &rewriteBatchedStatements=true.

          percent          live          alloc'ed  stack class
 rank   self  accum     bytes objs     bytes  objs trace name
    1 30.97% 30.97% 131523792 6431956 154628992 7460048 330688 byte[]
    2 16.37% 47.34%  69544416 859589  76181752 1002240 337650 byte[][]

Associated traces:

Responsible for 130MB of living objects after run is complete.

TRACE 330688:
	com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
	com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
	com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
	com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
	com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
	com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
	com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
	com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)

Responsible for 70MB of living objects after run is complete.

TRACE 337650:
	com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1361)
	com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2333)
	com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
	com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2040)
	com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1443)
	com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1777)
	com.mysql.jdbc.Connection.execSQL(Connection.java:3249)
	com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)