Bug #62007 Segfault in SQLDriverConnect ()
Submitted: 27 Jul 2011 19:43 Modified: 23 Sep 2013 12:14
Reporter: Christian Wyss Email Updates:
Status: Duplicate Impact on me:
None 
Category:Connector / ODBC Severity:S3 (Non-critical)
Version:5.1.8 OS:Linux (CentOS 5.4)
Assigned to: Bogdan Degtyariov CPU Architecture:Any

[27 Jul 2011 19:43] Christian Wyss
Description:
Crash after some time in multithreaded application with SIGSEGV (siginfo: address not mapped to object) while establishing connection to MySQL Server over ODBC:

#0  0x00002b15327b3a82 in my_stat () from /usr/lib64/libmyodbc5.so
(gdb) bt
#0  0x00002b15327b3a82 in my_stat () from /usr/lib64/libmyodbc5.so
#1  0x00002b15327b214f in ?? () from /usr/lib64/libmyodbc5.so
#2  0x00002b15327b2c90 in ?? () from /usr/lib64/libmyodbc5.so
#3  0x0000003a7140bd33 in pthread_once () from /lib64/libpthread.so.0
#4  0x00002b15327b2b30 in get_charset_by_csname () from /usr/lib64/libmyodbc5.so
#5  0x00002b15327cb0d2 in mysql_init_character_set () from /usr/lib64/libmyodbc5.so
#6  0x00002b15327cd0b8 in mysql_real_connect () from /usr/lib64/libmyodbc5.so
#7  0x00002b1532787ac7 in myodbc_do_connect () from /usr/lib64/libmyodbc5.so
#8  0x00002b1532788399 in MySQLDriverConnect () from /usr/lib64/libmyodbc5.so
#9  0x00002b1532780fa5 in SQLDriverConnect () from /usr/lib64/libmyodbc5.so
#10 0x0000003d0d4131f7 in SQLDriverConnect () from /usr/lib64/libodbc.so.1
#11 0x0000000000444b8c in CDBSession::connectDSN (this=0x22eb3680, szName=0x215f7668 "DSN=testDB;") at ../sysDBSession.cpp:207

CDBSession::connectDSN () function has at the time of crashing been previously called numerous times without problems.

odbc.ini declaration:

[testDB]
Driver       = /usr/lib64/libmyodbc5.so
Description  = MySql Database
SERVER       = 192.168.1.10
PORT         = 3306
USER         = MyUser
PASSWORD     = MyPass
DATABASE     = testDB
option       = 3
socket       =

Installed Packages
mysql-connector-odbc.x86_64 5.1.8-1.rhel5
unixODBC.x86_64 2.2.12-1.el5s2

Kernel 2.6.18-164.6.1.el5 x86_64

How to repeat:
Restart & wait
[29 Jul 2011 4:18] Bogdan Degtyariov
Christian,

Segfault in SQLDriverConnect() is not something we face every day, otherwise it would be already fixed.
Can you provide a C/C++ test case?
Thanks.
[4 Aug 2011 9:19] Christian Wyss
Hello Bogdan

It's difficult to provide a test case, as the crash seems to happen only after some time the process is running, and has allocated and discarded a multitude of the class handling the connection.

I will attach the c++ classes in use, but I suspect the crash being caused by either a therading issue where one thread deallocates something used in common, or a resource limit that is reached (open files, memory?) and not detected.

What I also did was installing the mysql-connector-odbc-debuginfo package before I went some days on holidays (that why it took so long to respond) - and after coming back I had another core file to analyze. That's the result:

#0  my_stat (path=0x78217020 "/usr/local/mysql/share/charsets/Index.xml", stat_area=0x78216f60, my_flags=0) at my_lib.c:540
540     my_lib.c: No such file or directory.
        in my_lib.c
(gdb) bt
#0  my_stat (path=0x78217020 "/usr/local/mysql/share/charsets/Index.xml", stat_area=0x78216f60, my_flags=0) at my_lib.c:540
#1  0x00002b2e4ef6814f in my_read_charset_file (filename=0x1 <Address 0x1 out of bounds>, myflags=0) at charset.c:336
#2  0x00002b2e4ef68c90 in init_available_charsets () at charset.c:429
#3  0x0000003a7140bd33 in pthread_once () from /lib64/libpthread.so.0
#4  0x00002b2e4ef68b30 in get_charset_by_csname (cs_name=0x25553b50 "utf8", cs_flags=32, flags=2015480128) at charset.c:570
#5  0x00002b2e4ef810d2 in mysql_init_character_set (mysql=0x2cf5d138) at client.c:1805
#6  0x00002b2e4ef830b8 in mysql_real_connect (mysql=0x2cf5d138, host=0x1e599a20 "192.168.1.10", user=0x28171920 "MyUser", passwd=0x2d921120 "MyPass", db=0x234f54a0 "testDB",
    port=3306, unix_socket=0x0, client_flag=131074) at client.c:2222
#7  0x00002b2e4ef3dac7 in myodbc_do_connect (dbc=0x2cf5d130, ds=0x26119f00) at connect.c:232
#8  0x00002b2e4ef3e399 in MySQLDriverConnect (hdbc=<value optimized out>, hwnd=0x0, szConnStrIn=0x26c548a0, cbConnStrIn=11, szConnStrOut=0x22a94ce0, cbConnStrOutMax=1024,
    pcbConnStrOut=0x7821bede, fDriverCompletion=0) at connect.c:802
#9  0x00002b2e4ef36fa5 in SQLDriverConnect (hdbc=0x2cf5d130, hwnd=0x0, in=<value optimized out>, in_len=<value optimized out>, out=0x7821bad0 "ect", out_max=<value optimized out>,
    out_len=0x7821bede, completion=0) at ansi.c:368
#10 0x0000003d0d4131f7 in SQLDriverConnect () from /usr/lib64/libodbc.so.1
#11 0x0000000000444b8c in CDBSession::connectDSN (this=0x2c3dd590, szName=0x1ef9b2c8 "DSN=testDB;") at ../sysDBSession.cpp:207
[11 Aug 2011 5:11] Bogdan Degtyariov
Christian,

Unfortunately, your C++ class is in no help to us because it only wraps ODBC calls to C++ classes. It does not show what exactly the program is doing.

However, I noticed you are using UnixODBC 2.2.14 64-bit, which has few ODBC types defined in an old-fashioned (32-bit) way. The recommended UnixODBC 2.2.14 fixes this problem, but the drivers built using headers/libraries version 2.2.14 are not completely compatible with UnixODBC 2.2.12.

Can you try running your project with UnixODBC 2.2.14?
[11 Aug 2011 7:49] Christian Wyss
Bogdan

Thought so. The whole thin gis basically a server application that answers SOAP calls, creating a handler thread for every request. 
Each handler thread has one of these Connection objects to communicate with the db.
As the issue only happens after some time and you need clients to get there the replication is quite difficult, and even if I could provide the full code you would need a pretty complicated test setup.

I'm a little confused by what you say about having noticed I am using UnixODBC 2.2.14 64-bit - as far as I can tell it should be 2.2.12, headers and libs.

I'll see if I can find some 2.2.14 rmps, but a quick search revealed that there is apparently nothing available for el5/CentOS5 - only for el6..

What I also just realized is that so far I've only seen this happen on virtualized boxes running a 2.6.18-164.6.1.el5xen kernel. Do you think this could be a possible reason?
[12 Aug 2011 7:50] Bogdan Degtyariov
Christian,

Sorry, my mistake. I mean you are using 2.2.12 and probably you should try 2.2.14. I don't know why 2.2.14 is not available for el5/CentOS5, but you can always build it from sources or use generic 64-bit binaries:

http://sourceforge.net/projects/unixodbc/files/unixODBC/2.2.14/
[12 Sep 2011 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[14 Nov 2011 9:37] Christian Wyss
Still occuring:

Program terminated with signal 11, Segmentation fault.
#0  my_stat (path=0x2aab12484020 "/usr/local/mysql/share/charsets/Index.xml", stat_area=0x2aab12483f60, my_flags=0) at my_lib.c:540
540     my_lib.c: No such file or directory.
        in my_lib.c
(gdb) bt
#0  my_stat (path=0x2aab12484020 "/usr/local/mysql/share/charsets/Index.xml", stat_area=0x2aab12483f60, my_flags=0) at my_lib.c:540
#1  0x00002b61e31f014f in my_read_charset_file (filename=0x1 <Address 0x1 out of bounds>, myflags=0) at charset.c:336
#2  0x00002b61e31f0c90 in init_available_charsets () at charset.c:429
#3  0x0000003d19c0bd33 in pthread_once () from /lib64/libpthread.so.0
#4  0x00002b61e31f0b30 in get_charset_by_csname (cs_name=0xe747fa0 "utf8", cs_flags=32, flags=306747712) at charset.c:570
#5  0x00002b61e32090d2 in mysql_init_character_set (mysql=0x101ef968) at client.c:1805
#6  0x00002b61e320b0b8 in mysql_real_connect (mysql=0x101ef968, host=0x1058a130 "192.168.85.10", user=0xe37bd10 "JobManager", passwd=0xfd7ade0 "JobManager", db=0xe68a3f0 "commDB",
    port=3306, unix_socket=0x0, client_flag=131074) at client.c:2222
#7  0x00002b61e31c5ac7 in myodbc_do_connect (dbc=0x101ef960, ds=0x107b65e0) at connect.c:232
#8  0x00002b61e31c6399 in MySQLDriverConnect (hdbc=<value optimized out>, hwnd=0x0, szConnStrIn=0xdf10110, cbConnStrIn=14, szConnStrOut=0xed5fa40, cbConnStrOutMax=1024,
    pcbConnStrOut=0x2aab12488ede, fDriverCompletion=0) at connect.c:802
#9  0x00002b61e31befa5 in SQLDriverConnect (hdbc=0x101ef960, hwnd=0x0, in=<value optimized out>, in_len=<value optimized out>, out=0x2aab12488ad0 "ect", out_max=<value optimized out>,
    out_len=0x2aab12488ede, completion=0) at ansi.c:368
#10 0x0000003d1a4131f7 in SQLDriverConnect () from /usr/lib64/libodbc.so.1
#11 0x0000000000449334 in CDBSession::connectDSN (this=0xecadbc0, szName=0x11cb96b8 "DSN=ums_pt_jm;") at ../sysDBSession.cpp:207
#12 0x0000000000449434 in CDBSession::connect (this=0xecadbc0) at ../sysDBSession.cpp:149
#13 0x000000000044964e in CDBSession (this=0xecadbc0, szName=0xda23768 "ums_pt_jm", szSessionId=0x59f861 "JobManagerServer") at ../sysDBSession.cpp:38
#14 0x000000000041cd45 in CJobManagerServer (this=0x2aab12489080, pSoap=0xdf0b370) at ../ccJobManagerServer.cpp:24
#15 0x000000000041a51d in CJobManager::pProcessServeLink (pSoap=0xdf0b370) at ../ccJobManager.cpp:156
#16 0x0000000000465df5 in CSoap::runThread (pData=0xdf0b370) at ../sysSoap.cpp:334
#17 0x0000003d19c064a7 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003d194d3c2d in clone () from /lib64/libc.so.6
Current language:  auto; currently c

Interesting: there is no /usr/local/mysql/share/charsets/Index.xml. Not even a /usr/local/mysql folder.
[23 Nov 2011 5:18] Bogdan Degtyariov
Christian,

Sorry for the delay and thanks for supplying the valuable details.
The Connector/ODBC driver is crashing when it tries to load the list of available charsets.

Setting to Verified.
[17 Apr 2012 7:48] Bogdan Degtyariov
Duplicate of bug 15547
[23 Sep 2013 11:53] Christian Wyss
I disagree with this being a duplicate of bug 15547.
That bug seems to describe a problem with latin1, but suggests the presence of the Index.xml file which is absent in my case.
[23 Sep 2013 12:10] Bogdan Degtyariov
Christian,

Thanks for your feedback.

Neither of the bug reports suggests having Index.xml file on your local machine.
It is about the initialization error, which makes libmysql (its binary code is embedded in the ODBC driver shared lib) to look for Index.xml. 
This file is not required and should not be tried to open.

Therefore, the duplicate status is correct.
[23 Sep 2013 12:14] Christian Wyss
Ok. Thank you, Bogdan.

I wonder about any news concerning the issue?
There does not have been much progress, I take?
[23 Sep 2013 12:28] Bogdan Degtyariov
Christian,

the problem is that we already fixed it several times, but it keeps coming under new conditions. Unfortunately, with 5.1.8 (and newer) we never succeeded in repeating the crash you mentioned.

Also, as I mentioned before, many OS distribute the ODBC driver dynamically linked with libmysqlclient.so, which we do not approve because of possible mismatches. 

We are working on putting extra checks to avoid errors during the initialization of libmysqlclient. This should improve the stability and can potentially resolve the error related to Index.xml. Though, there is no guarantee as we do not have any reliable test to reproduce the crash.