Bug #80621 Memory leak when using cursor(named_tuple=True)

• Submitted: 4 Mar 2016 23:29
• Modified: 28 Jul 2017 13:33
• Reporter: Piotr Jurkiewicz (OCA)
• Status: Closed
• Category: Connector / Python
• Severity: S2 (Serious)
• OS: Any
• CPU Architecture: Any
• Tags: Contribution

[4 Mar 2016 23:29] Piotr Jurkiewicz

Description:
Currently separate namedtuple class is created for each single row retrieved from DB. That is each single row has **its own** namedtuple class. Recreating of separate namedtuple classes not only reduces performance, but also results in higher memory usage and memory leaks.

How to repeat:
Run python application retrieving many rows using cursor(named_tuple=True) for couple of hours. You should see steady increase of RSS of python process.

I case of my application I observe growth at the rate of few MB/hour.

Suggested fix:
Rows having identical columns should share the same namedtuple class. This means that created namedtuple classes should be cached and reused when retrieving rows with identical columns, instead creating new class each time for each row.

I am going to attach a patch doing that.

I have been running mysql.connector with this patch for several months and haven't noticed any problems. Memory usage of python process with this patch is lower, and, more importantly, stays constant even in case of processes running continuously for several weeks.

[4 Mar 2016 23:32] Piotr Jurkiewicz

Patch adding caching of namedtuple classes

Attachment: named_tuple_cache.patch (text/x-patch), 1.02 KiB.

[4 Mar 2016 23:32] Piotr Jurkiewicz

Patch adding caching of namedtuple classes

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: named_tuple_cache.patch (text/x-patch), 1.02 KiB.

[5 Mar 2016 5:49] MySQL Verification Team

Hello Piotr Jurkiewicz,

Thank you for the report and contribution.

Thanks,
Umesh

[14 Oct 2016 8:20] Ondrej Medek

Please, fix this bug. MySQLCursorNamedTuple is unusable for a long running application, now. Or document the danger of the MySQLCursorNamedTuple.

The tupple cache may be per cursor or per connection.

[28 Jul 2017 13:33] Paul DuBois

Posted by developer:
 
Fixed in 2.1.7.

For cursors created with named_tuple=True, MySQLCursorNamedTuple
objects could leak memory. Thanks to Piotr Jurkiewicz for the patch
on which this fix was based.