Description:
I've written a program that is performing disk checks and is reporting its output to SQL using UPDATEs with string containing the check details, in an appending mode (the UPDATE is performed as: UPDATE checks_log SET log=concat(log,%new%data%))
The program is in C++ under Linux (using g++).
I'm using UNIXODBC version 2.3.0, The database is stored on a MySQL server version 5.0.77, and I've tried using BOTH of the following libs for the MyODBC connector:
libmyodbc3-3.51.26.so
libmyodbc5-5.1.6.so
I've tested this program both on my home Ubuntu 9.10 system, and both on my server (CentOS 5), and in BOTH i've had the same problem:
Because of the many updates being performed during, the system begins to CONSUME alot of memory and it doesn't stop!
I've tried to debug the code step by step and I have the following update function:
SQLRETURN databaseClass::executeQuery(SQLHANDLE sqlConn, const char *query)
{
SQLRETURN sqlReturn;
SQLHANDLE sqlStmt;
sqlReturn = SQLAllocHandle(SQL_HANDLE_STMT,sqlConn,&sqlStmt);
if (sqlReturn==SQL_SUCCESS || sqlReturn==SQL_SUCCESS_WITH_INFO)
{
sqlReturn = SQLExecDirect(sqlStmt,(SQLTCHAR*)query,strlen(query));
sqlReturn = SQLCloseCursor(sqlStmt);
sqlReturn = SQLFreeStmt(sqlStmt,SQL_CLOSE);
}
SQLFreeHandle(SQL_HANDLE_STMT,&sqlStmt);
return sqlReturn;
}
And everytime this function is being used to perform an UPDATE on the database, the memory consumption gets BIGGER and BIGGER and it doesn't stop!
At first I thought it may be a bug in my problem, but then I did a test where I use binary files to perform my updates (instead of writing the query to the DB, i just write them into a file), using this code:
if (use_files) {
if (fwrite(query,1,strlen(query),file)<5) {
error("couldn't write to file!");
}
} else Database.executeQuery(MySQLConnection,query);
(obviously, i set use_files=true)
And when I use files = NO MEMORY CONSUMPTION! program runs smoothly, no memory leaks and nothing.
So I assume this is probably a bug either in the MySQL ODBC Connector or UNIXODBC.
I've searched the UNIXODBC for ways to report this bug to them too but I haven't found how to do that yet, but once I do i'll report this bug to them as well.
Anyone familiar with this issue?
Thanks in advance,
Miki
How to repeat:
Install unixodbc v2.3.0 from code (It contains the header files "sql.h","sqlext.h" and "sqltypes.h" which are required for ODBC coding).
Install MySQL ODBC Connector from the latest version (I've used latest version).
Create a DSN for the MySQL on UNIXODBC using the MySQL ODBC library you installed. To do this, create some txt file (say, call it "myodbc.ini") with the following:
[MySQL]
Driver = /usr/lib/odbc/libmyodbc.so (<-- your myodbc lib's path!)
Description = MySQL_Database
SERVER = 127.0.0.1 (<-- your server's address)
PORT = 3306
USER = (your username)
Password = (your password)
Database =
OPTION = 3
And then add those details to UNIXODBC use:
odbcinst -i -s -f myodbc.ini (<-- your path to YOUR myodbc.ini)
On your DB, create a table with 2 columns: id (int - increasing), and textdata (text), and put 1000 predefined rows in it with SOME data.
THEN write a program in C++ under linux (using g++), that connects to the Database first (using SQLDriverConnect), and begins to perform APPENDING updates to this table using SQLExecDirect (each time it ADDS new text to the existing text on each row using: UPDATE table SET textdata=concat(textdata,'new data'))
If you'll run this program, you'll see that on every update the memory consumption is increased and never reduced
Description: I've written a program that is performing disk checks and is reporting its output to SQL using UPDATEs with string containing the check details, in an appending mode (the UPDATE is performed as: UPDATE checks_log SET log=concat(log,%new%data%)) The program is in C++ under Linux (using g++). I'm using UNIXODBC version 2.3.0, The database is stored on a MySQL server version 5.0.77, and I've tried using BOTH of the following libs for the MyODBC connector: libmyodbc3-3.51.26.so libmyodbc5-5.1.6.so I've tested this program both on my home Ubuntu 9.10 system, and both on my server (CentOS 5), and in BOTH i've had the same problem: Because of the many updates being performed during, the system begins to CONSUME alot of memory and it doesn't stop! I've tried to debug the code step by step and I have the following update function: SQLRETURN databaseClass::executeQuery(SQLHANDLE sqlConn, const char *query) { SQLRETURN sqlReturn; SQLHANDLE sqlStmt; sqlReturn = SQLAllocHandle(SQL_HANDLE_STMT,sqlConn,&sqlStmt); if (sqlReturn==SQL_SUCCESS || sqlReturn==SQL_SUCCESS_WITH_INFO) { sqlReturn = SQLExecDirect(sqlStmt,(SQLTCHAR*)query,strlen(query)); sqlReturn = SQLCloseCursor(sqlStmt); sqlReturn = SQLFreeStmt(sqlStmt,SQL_CLOSE); } SQLFreeHandle(SQL_HANDLE_STMT,&sqlStmt); return sqlReturn; } And everytime this function is being used to perform an UPDATE on the database, the memory consumption gets BIGGER and BIGGER and it doesn't stop! At first I thought it may be a bug in my problem, but then I did a test where I use binary files to perform my updates (instead of writing the query to the DB, i just write them into a file), using this code: if (use_files) { if (fwrite(query,1,strlen(query),file)<5) { error("couldn't write to file!"); } } else Database.executeQuery(MySQLConnection,query); (obviously, i set use_files=true) And when I use files = NO MEMORY CONSUMPTION! program runs smoothly, no memory leaks and nothing. So I assume this is probably a bug either in the MySQL ODBC Connector or UNIXODBC. I've searched the UNIXODBC for ways to report this bug to them too but I haven't found how to do that yet, but once I do i'll report this bug to them as well. Anyone familiar with this issue? Thanks in advance, Miki How to repeat: Install unixodbc v2.3.0 from code (It contains the header files "sql.h","sqlext.h" and "sqltypes.h" which are required for ODBC coding). Install MySQL ODBC Connector from the latest version (I've used latest version). Create a DSN for the MySQL on UNIXODBC using the MySQL ODBC library you installed. To do this, create some txt file (say, call it "myodbc.ini") with the following: [MySQL] Driver = /usr/lib/odbc/libmyodbc.so (<-- your myodbc lib's path!) Description = MySQL_Database SERVER = 127.0.0.1 (<-- your server's address) PORT = 3306 USER = (your username) Password = (your password) Database = OPTION = 3 And then add those details to UNIXODBC use: odbcinst -i -s -f myodbc.ini (<-- your path to YOUR myodbc.ini) On your DB, create a table with 2 columns: id (int - increasing), and textdata (text), and put 1000 predefined rows in it with SOME data. THEN write a program in C++ under linux (using g++), that connects to the Database first (using SQLDriverConnect), and begins to perform APPENDING updates to this table using SQLExecDirect (each time it ADDS new text to the existing text on each row using: UPDATE table SET textdata=concat(textdata,'new data')) If you'll run this program, you'll see that on every update the memory consumption is increased and never reduced