Bug #69004 called mysql_real_query is crash
Submitted: 19 Apr 2013 3:24 Modified: 20 Oct 2016 12:50
Reporter: dowflyon QIU Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: C API (client library) Severity:S3 (Non-critical)
Version:6.0.2 OS:Linux (ubuntu)
Assigned to: CPU Architecture:Any

[19 Apr 2013 3:24] dowflyon QIU
Description:
follow this code :
connect code:
{
    mysql_init(&st_hDatabaseMySQL.st_MySQL)
    mysql_options(&st_hDatabaseMySQL.st_MySQL,MYSQL_OPT_CONNECT_TIMEOUT,(char *)&nTimeOut)
    memcpy_s(&st_hDatabaseMySQL.st_ConnectInfo,sizeof(DATABASE_MYSQL_CONNECTINFO),pSt_MySQLConnector,sizeof(DATABASE_MYSQL_CONNECTINFO));
    mysql_real_connect(&st_hDatabaseMySQL.st_MySQL,pSt_MySQLConnector->tszHostAddr,pSt_MySQLConnector->tszName,pSt_MySQLConnector->tszPassWord,pSt_MySQLConnector->tszDBName,pSt_MySQLConnector->nPort,NULL,0)
    mysql_options(&st_hDatabaseMySQL.st_MySQL,MYSQL_OPT_RECONNECT,(char *)&bAutoReconnect)
    mysql_set_character_set(&st_hDatabaseMySQL.st_MySQL,"utf8")
    mysql_set_server_option(&st_hDatabaseMySQL.st_MySQL,MYSQL_OPTION_MULTI_STATEMENTS_ON)
    pthread_mutex_init(&st_hDatabaseMySQL.st_csDatabase,NULL);
    m_hNetComponents.NetEngine_LibHandleEx_Alloc(pxhData,st_hDatabaseMySQL)
    return TRUE;
}
query code:
{
    m_hNetComponents.NetEngine_LibHandleEx_Get(xhNet,&st_hDatabaseMySQL)
    pthread_mutex_lock(&st_hDatabaseMySQL.st_csDatabase);
    mysql_ping(&st_hDatabaseMySQL.st_MySQL);
    mysql_real_query(&st_hDatabaseMySQL.st_MySQL,lpszSQLQuery,_tcslen(lpszSQLQuery))
    pthread_mutex_unlock(&st_hDatabaseMySQL.st_csDatabase);
}

two dump core bt
#0  0xb779c424 in __kernel_vsyscall ()
#1  0xb752c1df in raise () from /lib/i386-linux-gnu/libc.so.6
#2  0xb752f825 in abort () from /lib/i386-linux-gnu/libc.so.6
#3  0xb756939a in ?? () from /lib/i386-linux-gnu/libc.so.6
#4  0xb7573ee2 in ?? () from /lib/i386-linux-gnu/libc.so.6
#5  0xb72e904e in my_no_flags_free (ptr=0x0) at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/mysys/my_malloc.c:65
#6  0xb72ca4a6 in end_server (mysql=0xb1d720b4) at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/libmysql/client.c:1033
#7  0xb72ca0fa in cli_safe_read (mysql=0xb1d720b4) at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/libmysql/client.c:688
#8  0xb72ccb36 in cli_read_query_result (mysql=0xb1d720b4) at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/libmysql/client.c:2825
#9  0xb72ccddc in mysql_real_query (mysql=0xb1d720b4, 
    query=0xb1d728bc "INSERT INTO `data_netflow_2013-04-18` (`host_ip`,`host_mac`,`up_flow`,`down_flow`,`http_up`,`http_down`,`http_total`,`smtp_up`,`smtp_down`,`smtp_total`,`pop3_up`,`pop3_down`,`pop3_total`,`ftp_up`,`ftp"..., length=510)
    at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/libmysql/client.c:2912
#10 0xb72c76d3 in CDataBase_MySQL::DataBase_MySQL_Execute (this=0xb7479f60 <m_MySQL>, xhNet=665, 
    lpszSQLQuery=0xb1d728bc "INSERT INTO `data_netflow_2013-04-18` (`host_ip`,`host_mac`,`up_flow`,`down_flow`,`http_up`,`http_down`,`http_total`,`smtp_up`,`smtp_down`,`smtp_total`,`pop3_up`,`pop3_down`,`pop3_total`,`ftp_up`,`ftp"...) at ./DataBase_MySql/DataBase_MySql.cpp:154
#11 0xb72c9d4a in DataBase_MySQL_Execute (xhData=665, 
    lpszSQLQuery=0xb1d728bc "INSERT INTO `data_netflow_2013-04-18` (`host_ip`,`host_mac`,`up_flow`,`down_flow`,`http_up`,`http_down`,`http_total`,`smtp_up`,`smtp_down`,`smtp_total`,`pop3_up`,`pop3_down`,`pop3_total`,`ftp_up`,`ftp"...) at ./DataBase_DllMain.cpp:45
(gdb) bt
#0  0xb779c424 in __kernel_vsyscall ()
#1  0xb752c1df in raise () from /lib/i386-linux-gnu/libc.so.6
#2  0xb752f825 in abort () from /lib/i386-linux-gnu/libc.so.6
#3  0xb756939a in ?? () from /lib/i386-linux-gnu/libc.so.6
#4  0xb7573ee2 in ?? () from /lib/i386-linux-gnu/libc.so.6
#5  0xb72e904e in my_no_flags_free (ptr=0x0) at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/mysys/my_malloc.c:65
#6  0xb72ca4a6 in end_server (mysql=0xb1d720b4) at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/libmysql/client.c:1033
#7  0xb72ca0fa in cli_safe_read (mysql=0xb1d720b4) at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/libmysql/client.c:688
#8  0xb72ccb36 in cli_read_query_result (mysql=0xb1d720b4) at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/libmysql/client.c:2825
#9  0xb72ccddc in mysql_real_query (mysql=0xb1d720b4, 
    query=0xb1d728bc "INSERT INTO `data_netflow_2013-04-18` (`host_ip`,`host_mac`,`up_flow`,`down_flow`,`http_up`,`http_down`,`http_total`,`smtp_up`,`smtp_down`,`smtp_total`,`pop3_up`,`pop3_down`,`pop3_total`,`ftp_up`,`ftp"..., length=510)
    at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/libmysql/client.c:2912
#10 0xb72c76d3 in CDataBase_MySQL::DataBase_MySQL_Execute (this=0xb7479f60 <m_MySQL>, xhNet=665, 
    lpszSQLQuery=0xb1d728bc "INSERT INTO `data_netflow_2013-04-18` (`host_ip`,`host_mac`,`up_flow`,`down_flow`,`http_up`,`http_down`,`http_total`,`smtp_up`,`smtp_down`,`smtp_total`,`pop3_up`,`pop3_down`,`pop3_total`,`ftp_up`,`ftp"...) at ./DataBase_MySql/DataBase_MySql.cpp:154
#11 0xb72c9d4a in DataBase_MySQL_Execute (xhData=665, 
    lpszSQLQuery=0xb1d728bc "INSERT INTO `data_netflow_2013-04-18` (`host_ip`,`host_mac`,`up_flow`,`down_flow`,`http_up`,`http_down`,`http_total`,`smtp_up`,`smtp_down`,`smtp_total`,`pop3_up`,`pop3_down`,`pop3_total`,`ftp_up`,`ftp"...) at ./DataBase_DllMain.cpp:45
#12 0xb773d651 in CLogicWork_Flow::LogicWork_Flow_InsertStorage (this=0xb7745030 <m_LogicFlow>, pSt_XMLFLow=0xb1d7312c) at ./Logic_ProWorkFlow/Logic_ProWorkFlow.cpp:55
#13 0xb7740fd5 in LogicWork_Flow_InsertStorage (pSt_XMLFLow=0xb1d7312c) at ./ProWork_DllMain.cpp:63

How to repeat:
always running mysql_real_query function

keep insert and updata sql Statement 

Our server is not very stable.

Every half an hour once crash....i don't know how to solve

Suggested fix:
i don't know
[19 Apr 2013 8:01] Shane Bester
the first step will be to compile your application with "-g -O1" gcc options, then run it in valgrind.  Find where the first invalid read/write or memory errors occur and solve them one by one :)

http://valgrind.org/
[22 Apr 2013 2:09] dowflyon QIU
can you see this len :my_no_flags_free (ptr=0x0)
  free NULL pointer,why?
[22 Apr 2013 2:45] dowflyon QIU
look this code.
exec SQL:INSERT INTO `data_netflow_2013-04-22` (`host_ip`,`host_mac`,`up_flow`,`down_flow`,`http_up`,`http_down`,`http_total`,`smtp_up`,`smtp_down`,`smtp_total`,`pop3_up`,`pop3_down`,`pop3_total`,`ftp_up`,`ftp_down`,`ftp_total`,`telnet_up`,`telnet_down`,`telnet_total`,`sql_up`,`sql_down`,`sql_total`,`other_up`,`other_down`,`other_total`,`log_time`)  VALUES('192.168.16.222','7845C423D1DA','394','0','0','0','0','0','0','0','0','0','0','0','0','0','0','0','0','0','0','0','416','0','416','1366596947').
exec sql error:Lost connection to MySQL server at 'reading initial communication packet', system error: 111

When executing the next SQL statement.my app is crash and print this.
*** glibc detected *** ./NetCenter_Service: double free or corruption (!prev): 0x08391960 ***

i am gdb into my app and Execution bt command
there is error: free NULL pointer?
#5  0xb72e904e in my_no_flags_free (ptr=0x0) at /export/home2/tmp/cteam/bs/connector-c-32bit/src/mysql-connector-c-6.0.2/mysys/my_malloc.c:65
[22 Apr 2013 5:17] Shane Bester
You do really need to run your application in valgrind to catch memory problems before they crash!
[23 Apr 2013 7:04] dowflyon QIU
yes i am test.look at all.what's mean

error:Lost connection to MySQL server during query,SQL:SELECT `rule_content` FROM `probe_status` WHERE `probe_code` = '000C2998F6ED'
==13407== Invalid read of size 4
==13407==    at 0x43E7C16: vio_close (viosocket.c:280)
==13407==    by 0x43E76ED: vio_delete (vio.c:226)
==13407==    by 0x43B5E99: end_server (client.c:1033)
==13407==    by 0x43B5AED: cli_safe_read (client.c:688)
==13407==    by 0x43B8529: cli_read_query_result (client.c:2825)
==13407==    by 0x43B87CF: mysql_real_query (client.c:2912)
==13407==    by 0x43B4312: CDataBase_MySQL::DataBase_MySQL_Execute(unsigned long long, char const*) (DataBase_MySql.cpp:152)
==13407==    by 0x4093300: Logic_ProWorkBt_InserStorage (ProWork_DllMain.cpp:56)
==13407==    by 0x40A8D99: CDetachLayer_Dispatch::DetachLayer_Dispatch_Beatherat(char*, char const*, int) (DetachLayer_Dispatch.cpp:156)
==13407==    by 0x40A8B9D: CDetachLayer_Dispatch::DetachLayer_Dispatch_Analyze(tag_NetCenter_Protocol_Header*, char const*, char*, int*) (DetachLayer_Dispatch.cpp:66)
==13407==    by 0x40AD5CC: DetachLayer_Dispatch_Analyze (DetachLayer_DllMain.cpp:64)
==13407==    by 0x804A617: NetCenter_PostTask(void*) (NetCenter_Task.cpp:101)
==13407==  Address 0x48b0358 is 0 bytes inside a block of size 384 free'd
==13407==    at 0x402BF06: free (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==13407==    by 0x43D4A41: my_no_flags_free (my_malloc.c:65)
==13407==    by 0x43B5E99: end_server (client.c:1033)
==13407==    by 0x43B5AED: cli_safe_read (client.c:688)
==13407==    by 0x43B8529: cli_read_query_result (client.c:2825)
==13407==    by 0x43B87CF: mysql_real_query (client.c:2912)
==13407==    by 0x43B473B: CDataBase_MySQL::DataBase_MySQL_ExecuteQuery(unsigned long long, char const*, unsigned long long*, unsigned long long*) (DataBase_MySql.cpp:224)
==13407== 
==13407== Warning: invalid file descriptor -1 in syscall close()
==13407== Invalid write of size 4
==13407==    at 0x43E7BE8: vio_close (viosocket.c:288)
==13407==    by 0x43E76ED: vio_delete (vio.c:226)
==13407==    by 0x43B5E99: end_server (client.c:1033)
==13407==    by 0x43B5AED: cli_safe_read (client.c:688)
==13407==    by 0x43B8529: cli_read_query_result (client.c:2825)
==13407==    by 0x43B87CF: mysql_real_query (client.c:2912)
==13407==    by 0x43B4312: CDataBase_MySQL::DataBase_MySQL_Execute(unsigned long long, char const*) (DataBase_MySql.cpp:152)
==13407==    by 0x4093300: Logic_ProWorkBt_InserStorage (ProWork_DllMain.cpp:56)
==13407==    by 0x40A8D99: CDetachLayer_Dispatch::DetachLayer_Dispatch_Beatherat(char*, char const*, int) (DetachLayer_Dispatch.cpp:156)
==13407==    by 0x40A8B9D: CDetachLayer_Dispatch::DetachLayer_Dispatch_Analyze(tag_NetCenter_Protocol_Header*, char const*, char*, int*) (DetachLayer_Dispatch.cpp:66)
==13407==    by 0x40AD5CC: DetachLayer_Dispatch_Analyze (DetachLayer_DllMain.cpp:64)
==13407==    by 0x804A617: NetCenter_PostTask(void*) (NetCenter_Task.cpp:101)
==13407==  Address 0x48b046c is 276 bytes inside a block of size 384 free'd
==13407==    at 0x402BF06: free (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==13407==    by 0x43D4A41: my_no_flags_free (my_malloc.c:65)
==13407==    by 0x43B5E99: end_server (client.c:1033)
==13407==    by 0x43B5AED: cli_safe_read (client.c:688)
==13407==    by 0x43B8529: cli_read_query_result (client.c:2825)
==13407==    by 0x43B87CF: mysql_real_query (client.c:2912)
==13407==    by 0x43B473B: CDataBase_MySQL::DataBase_MySQL_ExecuteQuery(unsigned long long, char const*, unsigned long long*, unsigned long long*) (DataBase_MySql.cpp:224)
[24 Apr 2013 6:31] dowflyon QIU
please note:
    This phenomenon exists only with the server instability 
i change mysql-server to local server,that's ok,Did not have any problems !
mysql server version :mysql  Ver 14.12 Distrib 5.0.95, for redhat-linux-gnu (i386) using readline 5.1
mysql connection c version is : 6.0.2

and have a question.
if i open the auto connection Options.then exec select or updata operator.mysql_error return an error.lost connection and mysql connection c should be auto connect server,But the server sometimes there should be too many connections is error.
[9 Jan 2014 12:30] Grzegorz Leszczyński
In our company we encountered the same problem, when moving from libmysqlclient16 to libmysqlclient18. After some debugging I found this is because we are using many statements (SQLAllocHandle(SQL_HANDLE_STMT)) for single connection and when query on one statement fails with connection error, queries on other statements fails, too, and it leads to memory corruption. I have similar output from Valgrind.

I'm not 100% sure that it not happens on libmysqlclient16, because we use only libmysqlclient18 on developer machines, which has slow connection to DB servers. But on production we use libmysqlclient16 and no crash ever happens, so I am almost sure, that such configuration works fine for libmysqlclient16.
[10 Jan 2014 16:21] Grzegorz Leszczyński
I forgot to add, that we are using libmysqlclient18 in version 5.5.34-0ubuntu0.12.04.1
[21 Jul 2014 19:52] Sveta Smirnova
Thank you for the report.

Please provide complete test case: table definitions, dump (if possible) and shortened C code which we can use to repeat the issue.
[23 Jul 2014 10:15] Grzegorz Leszczyński
It happens on any table and db.

To repeat that, just use connection with many statements and cut network connection to db (for eg. by using iptables). And try it few times, because it depends on race condition. I don't know how You want exact test case for it? You have to do it manually and program for connecting to DB is simple and You have it for sure :-) I don't have time for writing it myself and I'm not debugging it further, because we just stop using many statements in one connection, so it isn't an issue for us anymore.
[24 Jul 2014 16:32] Sveta Smirnova
Thank you for the feedback.

At first I want to ensure that your application uses mysql_real_query properly. Then in the initial description you mention that your network is not stable, but do not claim that crash happens, because client is disconnected from the server. Also I don't understand how " we just stop using many statements in one connection" related to network instability.
[24 Jul 2014 16:48] Grzegorz Leszczyński
>At first I want to ensure that your application uses mysql_real_query properly.

We are using mysqlclient via ODBC, not directly.

> Then in the initial description you mention that your network is not stable, but do not claim that crash happens, because client is disconnected from the server.

This is quote from my initial description:
"After some debugging I found this is because we are using many statements
(SQLAllocHandle(SQL_HANDLE_STMT)) for single connection and when query
on one statement fails with connection error, queries on other
statements fails, too, and it leads to memory corruption."

> Also I don't understand how " we just stop using many statements in one connection" related to network instability.

It is not related - network (or database) problems still happen, but they not lead to crash anymore, because we are using now single statement per connection (it is ODBC statement and I'm not sure if it is reflected in mysqlclient, so maybe bug is in ODBC).
[20 Oct 2016 12:50] Chiranjeevi Battula
Hello  dowflyon,

Thank you for your feedback.
I tried with latest version of MySQL C API versions as well and it worked without any issues.
Probably this is fixed in latest version, please feel free to reopen this bug report if you encounter this issue again.

Thanks,
Chiranjeevi.