Bug #69263 concurrent connect cause crash
Submitted: 17 May 2013 9:02 Modified: 25 Sep 2013 17:05
Reporter: china.ygw china.ygw Email Updates:
Status: Closed Impact on me:
None 
Category:Connector / ODBC Severity:S1 (Critical)
Version:5.2.5 OS:Linux
Assigned to: Bogdan Degtyariov CPU Architecture:Any
Tags: concurrent, connect, connector, crash, ODBC

[17 May 2013 9:02] china.ygw china.ygw
Description:
environment:
linux: cent os 6.2 32bit
mysql: v5.6.11
myodbc: v5.2.5
unixODBC: v 2.3.1

my program run as follow code:
open()
{
    SQLAllocHandle env handle and set some options.
    SQLAllocHandle hdb handle and set some options.
    SQLConnect database
}
close()
{
    SQLDisconnect database
    SQLFreeHandle hdb handle
    SQLFreeHandle env handle
}

when I continue to concurrent more than 500 connections less than 4 seconds, the program crash frequently. but, when I disable SQLConnect And SQLDisconnect code, every is ok.

when I use the c api, still crash.

How to repeat:
write some code as description.

Suggested fix:
I hope to fix as soon as possible.
[20 May 2013 8:48] Bogdan Degtyariov
Did I understand you correctly that a non-ODBC C program that uses pure MySQL C API crashes as well?
[20 May 2013 9:07] Bogdan Degtyariov
One more question: is your program a multithreaded one? Does it open one connection per thread or all connections are being open in one thread?
Thanks.
[20 May 2013 13:19] china.ygw china.ygw
crash 1

Attachment: 1.txt (text/plain), 32.14 KiB.

[20 May 2013 13:19] china.ygw china.ygw
crash 2

Attachment: 2.txt (text/plain), 1.74 KiB.

[20 May 2013 13:19] china.ygw china.ygw
crash 3

Attachment: 3.txt (text/plain), 1.86 KiB.

[20 May 2013 13:20] china.ygw china.ygw
crash 4

Attachment: 4.txt (text/plain), 1.56 KiB.

[20 May 2013 13:20] china.ygw china.ygw
crash 5

Attachment: 5.txt (text/plain), 8.98 KiB.

[20 May 2013 13:21] china.ygw china.ygw
odbc.ini:
[dsn__mysql__test]
Description     = mysql
Driver             = mysql
Server            = localhost
Port                = 3306
Socket            =
Database        = test
User                =
Password        =

odbcinst.ini:
[mysql]
Description     = mysql
Driver              = /myodbc/lib/libmyodbc5w.so
Threading       = 0

my test code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unixODBC/sql.h>
#include <unixODBC/sqlext.h>
#include <pthread.h>
#include <semaphore.h>

#define THREAD_COUNT   500

sem_t       sem;

typedef struct _tinfo {
    int id;
    char *dsn;
    char *uid;
    char *pwd;
} tinfo;

void *thread( void *c )
{
tinfo *ti = (tinfo *)c;
SQLHANDLE    hEnv;
SQLHANDLE hDbc;
SQLHANDLE hStmt;
SQLRETURN ret;
int i, count;

    for ( i = 0; i < 100; i ++ ) {

        ret = SQLAllocHandle( SQL_HANDLE_ENV, SQL_NULL_HANDLE, &hEnv );

        ret = SQLSetEnvAttr( hEnv, SQL_ATTR_ODBC_VERSION, (SQLPOINTER)SQL_OV_ODBC3, SQL_IS_UINTEGER );
       
        ret = SQLAllocHandle( SQL_HANDLE_DBC, hEnv, &hDbc );
       
        ret = SQLConnect( hDbc, (SQLCHAR*)ti -> dsn, SQL_NTS, (SQLCHAR*)ti -> uid, SQL_NTS,
            (SQLCHAR*)ti -> pwd, SQL_NTS);

        printf( "%d:%d %d\n", ti -> id, i, ret );

        SQLDisconnect( hDbc );
        SQLFreeHandle( SQL_HANDLE_DBC, hDbc );
        SQLFreeHandle( SQL_HANDLE_ENV, hEnv );

    }

    printf( "finished set\n" );

    sem_post(&sem);

    return NULL;
}

main( int argc, char **argv )
{
tinfo ti[ THREAD_COUNT ];
int i;
pthread_t   th[ THREAD_COUNT ];

    if ( argc != 4 ) {
        fprintf( stderr, "usage: ts1 dsn user pass\n" );
        exit( -1 );
    }

    sem_init(&sem, 0, -THREAD_COUNT);

    for ( i = 0; i < THREAD_COUNT; i ++ ) {

        ti[ i ].dsn = argv[ 1 ];
        ti[ i ].uid = argv[ 2 ];
        ti[ i ].pwd = argv[ 3 ];
        ti[ i ].id = i;

        pthread_create( th + i, NULL, thread, &ti[ i ] );
    }

    sem_wait(&sem);
}
[23 May 2013 8:33] Bogdan Degtyariov
Thank you for reporting the problem in MySQL Software and for providing the test case, which allowed to repeat it. It was a bit tricky because as I had to optimize my Virtual Machine to gain the needed rate of 400 connections per second.

Setting the bug status to Verified.
[27 May 2013 7:24] Bogdan Degtyariov
The problem is in libmysql, so it was inherited by Connector/ODBC because it is based on libmysql. The stack is as follows:

Program terminated with signal 11, Segmentation fault.
#0  my_stat (path=0x7f306b744d80 "/usr/share/mysql/charsets/Index.xml", stat_area=0x7f306b744c60, my_flags=0)
    at /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/mysys/my_lib.c:423
423	/pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/mysys/my_lib.c: No such file or directory.
(gdb) where
#0  my_stat (path=0x7f306b744d80 "/usr/share/mysql/charsets/Index.xml", stat_area=0x7f306b744c60, my_flags=0)
    at /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/mysys/my_lib.c:423
#1  0x0000000000422a93 in my_read_charset_file (loader=0x7f306b744f90, filename=0x0, myflags=1802791440)
    at /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/mysys/charset.c:420
#2  0x0000000000423357 in init_available_charsets () at /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/mysys/charset.c:508
#3  0x00007f306c734400 in pthread_once () from /lib/x86_64-linux-gnu/libpthread.so.0
#4  0x0000000000423c2c in my_charset_get_by_name (loader=0x7f306b7452f0, cs_name=0x7f3058000980 "utf8", cs_flags=1802791440, flags=1803735557)
    at /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/mysys/charset.c:731
#5  0x0000000000423d33 in get_charset_by_csname (cs_name=0x7f3058000980 "utf8", cs_flags=32, flags=16)
    at /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/mysys/charset.c:752
#6  0x000000000040aecf in mysql_set_character_set_with_default_collation (mysql=<optimised out>)
    at /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sql-common/client.c:2156
#7  mysql_init_character_set (mysql=0x7f306b745970) at /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sql-common/client.c:2195
#8  0x000000000040e8d0 in mysql_real_connect (mysql=0x7f306b745970, host=0x49d430 "localhost", user=0x7fff7e0c247e "dbs2", passwd=0x7fff7e0c2483 "dbspass", db=0x49d28d "test", port=0, 
    unix_socket=0x49d29c "/tmp/mysql.sock", client_flag=0) at /pb2/build/sb_0-9099207-1367228139.78/rpm/BUILD/mysql-5.6.11/mysql-5.6.11/sql-common/client.c:3616
#9  0x0000000000409795 in do_connect(char*, char*) ()
#10 0x000000000040984d in thread(void*) ()
#11 0x00007f306c72ee9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007f306b83acbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000000000 in ?? ()
[27 May 2013 7:36] Bogdan Degtyariov
Can you try compiling your test program with -pthread option?
Libmysql uses pthread_once() function to initialize internal charsets array in multithreaded environment. The problem could be caused by build command which does not include -pthread option. Without this option linker uses pthread_once() implementation from libc.

The problem is not repeatable each time, so I had to do several run attempts to repeat the crash. After using -pthread option the crashes seemed to stop.
Can you try -pthread and let me know whether it helped for you as well?
[27 May 2013 13:25] china.ygw china.ygw
I am sorry to see that the problem still open when I use the -lpthread option.

the new crash information can be found attachments.
[27 May 2013 13:42] china.ygw china.ygw
new crash 1 for -lpthread option

Attachment: 111.txt (text/plain), 7.11 KiB.

[27 May 2013 13:42] china.ygw china.ygw
new crash 2 for -lpthread option

Attachment: 222.txt (text/plain), 1.16 KiB.

[27 May 2013 13:42] china.ygw china.ygw
new crash 3 for -lpthread option

Attachment: 333.txt (text/plain), 15.68 KiB.

[27 May 2013 15:34] Bogdan Degtyariov
Thanks for your feedback, but I asked to try -pthread option instead of -lpthread.
[29 May 2013 4:03] china.ygw china.ygw
the crash still open when I use the -pthread option.
some crash information see attachments.
[29 May 2013 4:03] china.ygw china.ygw
crash 1 with -pthread option.

Attachment: 1111.txt (text/plain), 1.78 KiB.

[29 May 2013 4:03] china.ygw china.ygw
crash 2 with -pthread option.

Attachment: 2222.txt (text/plain), 1.96 KiB.

[29 May 2013 4:04] china.ygw china.ygw
crash 3 with -pthread option.

Attachment: 3333.txt (text/plain), 8.20 KiB.

[30 May 2013 11:29] Bogdan Degtyariov
Thank you for sending extra details about the crash.
In the last file you uploaded (333333.txt) I can see this:

/mysql/lib/libmysqlclient.so.18(my_malloc+0x2f)[0x54313f]

In other words, your version of Connector/ODBC driver has been linked against the shared version of libmysqlclient.so, which is not right. Our release versions are linked using the static library (libmysqlclient_r.a), so mysql client code is embedded into the driver binary file. This is done for stability purposes and to make sure the mysql client library is not missing if installed in a non-standard directory.

The easiest way of linking against the .a version of libmysqlclient is to temporarily remove the libmysqlclient.so*.

Also, it seems that UnixODBC has issues with the threads isolation. The usual practice with allocating environment (SQLHENV) handler is to do it just once and then allocate all connection (SQLHDBC) handlers using just this SQLHENV.

I changed the test case by declaring henv a global variable, which is allocated in main() and then this handle is shared between threads. This not only increased the number of connections per second, but also completely removed crashes.
[27 Jun 2013 15:16] china.ygw china.ygw
in my 5.5.16 mysql, libmysqlclient_r.so.18 is a soft link to libmysqlclient.so.

I compile mysql with follow cmake options:
cmake .. -DCMAKE_INSTALL_PREFIX=/mysql -DMYSQL_UNIX_ADDR=/tmp/mysql.sock -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DWITH_EXTRA_CHARSETS=all -DENABLED_LOCAL_INFILE=1

could you tell me how to compile the real libmysqlclient_r.so? I use google and can not find the answer.

when I connect to another database (such as sqlserver), everything is ok.
[1 Jul 2013 7:04] Bogdan Degtyariov
As I have already explained, your driver should not depend on libmysqlclient_r.so at all. Also, I explained how to link against the static version (libmysqlclient_r.a).

Have you tried the binary packages with Connector/ODBC distributed by Oracle?
These packages contain ODBC drivers that do not depend on libmysqlclient_r.so.
[2 Aug 2013 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[2 Aug 2013 5:25] Bogdan Degtyariov
Please note that the current bug report has been waiting on your feedback.
Can you please try the standard ODBC driver built by Oracle?
It can be downloaded from here:

http://dev.mysql.com/downloads/connector/odbc/

Thanks.
[3 Sep 2013 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[25 Sep 2013 17:05] china.ygw china.ygw
thanks very much.
when I use the binary packages with MySQL Connector/ODBC, and allocating environment (SQLHENV) handler just once, every is ok.