Bug #74696 Django backend: creating too many connections per request
Submitted: 5 Nov 2014 8:10 Modified: 3 Dec 2014 15:28
Reporter: Jan Vorcak Email Updates:
Status: Closed Impact on me:
None 
Category:Connector / Python Severity:S2 (Serious)
Version:2.0.2 OS:Linux
Assigned to: CPU Architecture:Any
Tags: Django, python

[5 Nov 2014 8:10] Jan Vorcak
Description:
Using Django==1.7.1, python==3.3.2 and mysql-connector-python==2.0.2

1) mysql-connector-python creates too many mysql connections
2) that also results in non functional CONN_MAX_AGE setting, https://docs.djangoproject.com/en/1.7/ref/settings/#conn-max-age

How to repeat:
from django.db.backends.signals import connection_created

def connection_setup(**kwargs):
    conn = kwargs['connection']
    print("Created mysql connection", conn)

connection_created.connect(connection_setup)

Once you do that and load for instance a /admin/user/<id>/ you'll get following

Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
[05/Nov/2014 09:01:35] "GET /admin/auth/user/5/ HTTP/1.1" 200 16548
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158378250>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158378cd0>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158316090>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51583354d0>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f515837a090>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158363a90>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158378cd0>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158378250>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158316090>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51583354d0>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158363a90>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f515837a090>
[05/Nov/2014 09:01:36] "GET /static/admin/js/admin/RelatedObjectLookups.js HTTP/1.1" 404 119
[05/Nov/2014 09:01:36] "GET /static/admin/css/forms.css HTTP/1.1" 404 100
[05/Nov/2014 09:01:36] "GET /static/admin/js/jquery.min.js HTTP/1.1" 404 103
[05/Nov/2014 09:01:36] "GET /static/admin/css/base.css HTTP/1.1" 404 99
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158324ed0>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158324910>
[05/Nov/2014 09:01:36] "GET /static/admin/js/core.js HTTP/1.1" 404 97
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582873d0>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582e8490>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582ff350>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158324910>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f5158324ed0>
[05/Nov/2014 09:01:36] "GET /admin/jsi18n/ HTTP/1.1" 200 2372
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582ffd10>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582e8490>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582ff350>
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582873d0>
[05/Nov/2014 09:01:36] "GET /static/admin/js/actions.min.js HTTP/1.1" 404 104
[05/Nov/2014 09:01:36] "GET /static/admin/js/jquery.init.js HTTP/1.1" 404 104
[05/Nov/2014 09:01:36] "GET /static/admin/js/SelectBox.js HTTP/1.1" 404 102
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582ffd10>
[05/Nov/2014 09:01:36] "GET /static/admin/js/SelectFilter2.js HTTP/1.1" 404 106
[05/Nov/2014 09:01:36] "GET /static/admin/js/calendar.js HTTP/1.1" 404 101
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582a4e50>
[05/Nov/2014 09:01:36] "GET /static/admin/js/admin/DateTimeShortcuts.js HTTP/1.1" 404 116
Created mysql connection <mysql.connector.django.base.DatabaseWrapper object at 0x7f51582a4e50>
[05/Nov/2014 09:01:36] "GET /static/admin/img/icon_addlink.gif HTTP/1.1" 404 107

while with http://github.com/clelland/MySQL-for-Python-3.git you get

Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Created mysql connection <django.db.backends.mysql.base.DatabaseWrapper object at 0x7f10230e0690>
[05/Nov/2014 09:02:05] "GET /admin/auth/user/5/ HTTP/1.1" 200 25875
[05/Nov/2014 09:02:05] "GET /static/admin/js/admin/RelatedObjectLookups.js HTTP/1.1" 200 3232
[05/Nov/2014 09:02:05] "GET /static/admin/js/jquery.js HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/js/core.js HTTP/1.1" 200 6899
[05/Nov/2014 09:02:05] "GET /static/admin/css/forms.css HTTP/1.1" 200 5928
[05/Nov/2014 09:02:05] "GET /static/admin/js/actions.js HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/css/base.css HTTP/1.1" 200 13995
[05/Nov/2014 09:02:05] "GET /static/admin/js/jquery.init.js HTTP/1.1" 200 326
[05/Nov/2014 09:02:05] "GET /static/debug_toolbar/css/toolbar.css HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/js/calendar.js HTTP/1.1" 200 5969
[05/Nov/2014 09:02:05] "GET /static/admin/js/SelectFilter2.js HTTP/1.1" 200 9865
[05/Nov/2014 09:02:05] "GET /static/admin/js/admin/DateTimeShortcuts.js HTTP/1.1" 200 17105
[05/Nov/2014 09:02:05] "GET /static/admin/js/SelectBox.js HTTP/1.1" 200 4222
[05/Nov/2014 09:02:05] "GET /static/debug_toolbar/js/toolbar.js HTTP/1.1" 304 0
Created mysql connection <django.db.backends.mysql.base.DatabaseWrapper object at 0x7f102156bd50>
[05/Nov/2014 09:02:05] "GET /static/admin/css/widgets.css HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/img/icon_addlink.gif HTTP/1.1" 200 119
[05/Nov/2014 09:02:05] "GET /static/debug_toolbar/img/ajax-loader.gif HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /admin/jsi18n/ HTTP/1.1" 200 2372
[05/Nov/2014 09:02:05] "GET /static/admin/img/nav-bg-reverse.gif HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/img/default-bg.gif HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/img/nav-bg.gif HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/img/icon_deletelink.gif HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/img/icon_calendar.gif HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/img/icon_clock.gif HTTP/1.1" 304 0
[05/Nov/2014 09:02:05] "GET /static/admin/img/selector-icons.gif HTTP/1.1" 200 2771
[05/Nov/2014 09:02:05] "GET /static/admin/img/icon-unknown.gif HTTP/1.1" 200 130
[05/Nov/2014 09:02:05] "GET /static/admin/img/selector-search.gif HTTP/1.1" 200 552
[6 Nov 2014 6:23] Peeyush Gupta
Verified.

Everytime a DatabaseWrapper object is created, Connector/Python Django backend is
creating a connection. Whereas the other backend does not make a connection to the server on creating DatabaseWrapper object right away and create the connection only when its needed.
But the no. of DatabaseWrapper Objects are same for both the backends.
Also IMHO CONN_MAX_AGE still works as the database connections are closed as soon as CONN_MAX_AGE is over.

Since the backend is not leaking connections, I'm changing the severity to S2.
[3 Dec 2014 15:28] Paul DuBois
Noted in 2.0.3 changelog.

The Django backend was creating excessive connections (immediately
when each DatabaseWrapper object was created rather than waiting
until the object actually needed the connection.)
[30 Mar 2015 16:01] Paul DuBois
Noted in 2.1.2 changelog.