Bug #99517 Method executemany has a memory leak when include Decimal python data type

• Submitted: 12 May 2020 1:46
• Modified: 31 Jul 2020 22:24
• Reporter: kan liyong (OCA)
• Status: Closed
• Category: Connector / Python
• Severity: S2 (Serious)
• Version: 8.0.20
• OS: MacOS
• CPU Architecture: Any
• Tags: connecotor python, memory leak

[12 May 2020 1:46] kan liyong

Description:
Memory leak when inserting large amount of data using mysql-connector-python

Memory grows consistently until out-of-memory

In the end I found that this problem only occurs when the Decimal attribute is included

How to repeat:
files:

1: Python https://github.com/kanliyong/bugs/blob/master/eqs_share_order_test.py
2: SQL https://github.com/kanliyong/bugs/blob/master/eqs_share_order_test.sql

Step 1:
    import the SQL file into Mysql database Server (5.7)
   mysql >>> source /path/to/sql_file
Step 2:
    Run python scripts
   $ python3 eqs_share_order_test.py
Step 3:
    monitor Memory usages:
   $ top

Suggested fix:
Possible problems:

PyObject*
MySQL_convert_to_mysql(MySQL *self, PyObject *args)

PyObject*
pytomy_decimal(PyObject *obj)

[12 May 2020 8:22] MySQL Verification Team

Hello kanly kan,

Thank you for the report and feedback.

regards,
Umesh

[13 May 2020 0:55] kan liyong

eqs_share_order_test.py

Attachment: eqs_share_order_test.py (text/x-python-script), 1.54 KiB.

[15 May 2020 6:37] kan liyong

I seem to have solved this problem！！

After my test, the problem does not appear。

Change the code as follows ...

mysql_capi_conversion.c

PyObject*
pytomy_decimal(PyObject *obj)
{
#ifdef PY3
    PyObject *str = PyObject_Str(obj);
    PyObject *ret = (const char *)PyUnicode_1BYTE_DATA(str);
    Py_DECREF(str);
    return ret;
#else
    PyObject *numeric, *new_num;

[15 May 2020 6:52] kan liyong

PyObject*
pytomy_decimal(PyObject *obj)
{
#ifdef PY3
    PyObject *str = PyObject_Str(obj);
    PyObject *ret = (const char *)PyUnicode_1BYTE_DATA(str);
    Py_DECREF(str);
    return PyBytes_FromString(ret);
#else

[15 May 2020 7:37] kan liyong

I have submitted PR:
https://github.com/mysql/mysql-connector-python/pull/60

PyObject*
pytomy_decimal(PyObject *obj)
{
#ifdef PY3
    PyObject *str = PyObject_Str(obj);
    PyObject *ret_tmp = (const char *)PyUnicode_1BYTE_DATA(str);
    PyObject *ret = PyBytes_FromString(ret_tmp);
    Py_DECREF(ret_tmp);
    return ret;
#else

[26 May 2020 13:59] OCA Admin

Contribution submitted via Github - Bug fix : memory leak when include Decimal python data type 
(*) Contribution by kan lying (Github kanliyong, mysql-connector-python/pull/61#issuecomment-634038352): I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: git_patch_418456870.txt (text/plain), 2.07 KiB.

[14 Jul 2020 2:53] kan liyong

In the new version 8.0.21, this issue is not resolved ?

[31 Jul 2020 22:24] Philip Olson

Posted by developer:
 
Fixed as of the upcoming MySQL Connector/Python 8.0.22 release, and here's the proposed changelog entry from the documentation team:

Fixed a memory leak in the C-extension implementation when using the
Decimal data type. Thanks to Kan Liyong for the patch.

Thank you for the bug report.

[20 Oct 2020 6:56] Frederic Descamps

Thank you for your contribution that has been added to 8.0.22: https://lefred.be/content/mysql-8-0-22-thank-you-for-the-contributions/