Bug #98457 Crash using mysql_real_connect
Submitted: 1 Feb 2020 12:24 Modified: 10 Feb 2020 13:50
Reporter: Adrián G. Email Updates:
Status: Not a Bug Impact on me:
None 
Category:Connector / C Severity:S1 (Critical)
Version:5.7.27 OS:Ubuntu (18.04.1)
Assigned to: CPU Architecture:x86 (x86_64)
Tags: linux, multithreaded, mysql_real_connect

[1 Feb 2020 12:24] Adrián G.
Description:
Hi, the following crash seems to happen sometimes at mysql_real_connect.

Call is performed inside code from a custom MySQL C wrapper interface library that's loaded dynamically (dlsym/dlopen) by a game server main library.

It seems it's caused by malloc. What potential causes could incur into it crashing with a SIGABRT signal?

Thanks

----------------------------------------------
CRASH: Sat Feb  1 12:16:33 CET 2020
Start Line: ./srcds_linux -game hl2mp -debug -dumplongticks +maxplayers 32 +mp_teamplay 1 +map rp_c18_v1
[New LWP 7089]
[New LWP 7075]
[New LWP 7076]
[New LWP 7077]
[New LWP 7078]
[New LWP 7079]
[New LWP 7080]
[New LWP 7081]
[New LWP 7082]
[New LWP 7086]
[New LWP 7087]
[New LWP 7088]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./srcds_linux -game hl2mp -debug -dumplongticks +maxplayers 32 +mp_teamplay 1 +'.
Program terminated with signal SIGABRT, Aborted.
#0  0xf7f9d939 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0xe9ff7b40 (LWP 7089))]
#0  0xf7f9d939 in __kernel_vsyscall ()
#1  0xf7be3832 in raise () from /lib/i386-linux-gnu/libc.so.6
#2  0xf7be4cc1 in abort () from /lib/i386-linux-gnu/libc.so.6
#3  0xf7c2c66b in ?? () from /lib/i386-linux-gnu/libc.so.6
#4  0xf7c2fc70 in ?? () from /lib/i386-linux-gnu/libc.so.6
#5  0xf7c30e38 in malloc () from /lib/i386-linux-gnu/libc.so.6
#6  0xe9b23fc3 in my_raw_malloc (my_flags=16, size=<optimized out>) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/mysys/my_malloc.c:191
#7  my_malloc (key=0, size=396, flags=16) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/mysys/my_malloc.c:54
#8  0xe9b2e7c0 in mysql_socket_vio_new (mysql_socket=..., type=VIO_TYPE_TCPIP, flags=2) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/vio/vio.c:274
#9  0xe9b2eaa3 in vio_new (sd=27, type=VIO_TYPE_TCPIP, flags=2) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/vio/vio.c:293
#10 0xe9b04477 in mysql_real_connect (mysql=0xb0bcd0c, host=0xb6d6a90 "remotemysql.com", user=0xb6d64d0 "<my_username>", passwd=0xb6d63b0 "<my_password>", db=0xb6d6280 "fSkSY53EP5", port=3306, unix_socket=0x0, client_flag=0) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/sql-common/client.c:4277
#11 0xe9ace98d in CMySQLConnection::Connect (this=0xb0bcd00, pHostName=0xb6d6a90 "remotemysql.com", pUserName=0xb6d64d0 "<my_username>", pPassword=0xb6d63b0 "<my_password>", pSchemaName=0xb6d6280 "fSkSY53EP5", port=3306) at /home/vsts/work/1/s/mp/src/game/server/hl2rp/dal/mysql_driver/mysql_driver.cpp:344
#12 0xf1e0b4c4 in CDAL::LoadDatabaseConfiguration (this=0xf2667840 <gDAL>) at /home/vsts/work/1/s/mp/src/game/server/hl2rp/dal/dal.cpp:320
#13 0xf1e0aec9 in CDAL::Run (this=0xf2667840 <gDAL>) at /home/vsts/work/1/s/mp/src/game/server/hl2rp/dal/dal.cpp:208
#14 0xf7b82690 in CThread::ThreadProc(void*) () from bin/libtier0_srv.so
#15 0xf7d983bd in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#16 0xf7caee16 in clone () from /lib/i386-linux-gnu/libc.so.6
No symbol table info available.
eax            0x0      0
ecx            0xe9ff688c       -369137524
edx            0x0      0
ebx            0x2      2
esp            0xe9ff6870       0xe9ff6870
ebp            0xe9ff688c       0xe9ff688c
esi            0x8      8
edi            0x0      0
eip            0xf7f9d939       0xf7f9d939 <__kernel_vsyscall+9>
eflags         0x286    [ PF SF IF ]
cs             0x23     35
ss             0x2b     43
ds             0x2b     43
es             0x2b     43
fs             0x3      3
gs             0x63     99
k0             0x0      0
k1             0x0      0
k2             0x0      0
k3             0x0      0
k4             0x0      0
k5             0x0      0
k6             0x0      0
k7             0x0      0
From        To          Syms Read   Shared Object Library
0xf7e92590  0xf7f528f5  Yes (*)     /lib/i386-linux-gnu/libm.so.6
0xf7e84ad0  0xf7e85a84  Yes (*)     /lib/i386-linux-gnu/libdl.so.2
0xf7df7914  0xf7e3dc78  Yes         bin/libstdc++.so.6
0xf7d969d0  0xf7da620f  Yes (*)     /lib/i386-linux-gnu/libpthread.so.0
0xf7bce610  0xf7d1b386  Yes (*)     /lib/i386-linux-gnu/libc.so.6
0xf7f9eab0  0xf7fb97fb  Yes (*)     /lib/ld-linux.so.2
0xf7ba1e00  0xf7bb10fc  Yes         bin/libgcc_s.so.1
0xf7b7b1d0  0xf7b8e44c  Yes (*)     bin/libtier0_srv.so
0xf7b68910  0xf7b6c3a4  Yes (*)     /lib/i386-linux-gnu/librt.so.1
0xf7a6eec0  0xf7a9d944  Yes (*)     bin/libvstdlib_srv.so
0xf7435460  0xf75a6a10  Yes (*)     bin/dedicated_srv.so
0xf73bc4e0  0xf73cad3c  Yes (*)     bin/libsteam_api.so
0xf7290e50  0xf729cd44  Yes (*)     /lib32/libtinfo.so.5
0xf676ddd0  0xf692d208  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/engine_srv.so
0xf6287d30  0xf62ad3b4  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/soundemittersystem_srv.so
0xf613b1c0  0xf620e324  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/materialsystem_srv.so
0xf3c84cf0  0xf3d2f608  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/studiorender_srv.so
0xf3ae4590  0xf3c03860  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/vphysics_srv.so
0xf3a47a20  0xf3a92ee8  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/datacache_srv.so
0xf3a0d190  0xf3a21cbc  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/shaderapiempty_srv.so
0xf1831f20  0xf20e44e4  Yes         /home/steam/.steam/steamcmd/hl2rp_server/hl2mp/bin/server_srv.so
0xf38a0630  0xf38b5a3c  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/scenefilecache.so
0xec2cc880  0xed4d2fb4  Yes (*)     /home/steam/.steam/sdk32/steamclient.so
0xf3844bb0  0xf384b554  Yes (*)     /lib/i386-linux-gnu/libnss_files.so.2
0xf3a01b70  0xf3a04ad4  Yes (*)     /lib/i386-linux-gnu/libnss_dns.so.2
0xf382d660  0xf38394c4  Yes (*)     /lib/i386-linux-gnu/libresolv.so.2
0xeb079440  0xebc547c0  Yes (*)     bin/steamclient.so
0xeaa74170  0xeaa9c0a4  Yes (*)     bin/crashhandler.so
0xe9acd920  0xe9ba1088  Yes         /home/steam/.steam/steamcmd/hl2rp_server/bin/mysql_driver.so
(*): Shared library is missing debugging information.
Dump of assembler code for function __kernel_vsyscall:
   0xf7f9d930 <+0>:     push   %ecx
   0xf7f9d931 <+1>:     push   %edx
   0xf7f9d932 <+2>:     push   %ebp
   0xf7f9d933 <+3>:     mov    %esp,%ebp
   0xf7f9d935 <+5>:     sysenter
   0xf7f9d937 <+7>:     int    $0x80
=> 0xf7f9d939 <+9>:     pop    %ebp
   0xf7f9d93a <+10>:    pop    %edx
   0xf7f9d93b <+11>:    pop    %ecx
   0xf7f9d93c <+12>:    ret
End of assembler dump.
Stack level 0, frame at 0xe9ff6880:
 eip = 0xf7f9d939 in __kernel_vsyscall; saved eip = 0xf7be3832
 called by frame at 0xe9ff69b0
 Arglist at 0xe9ff6878, args:
 Locals at 0xe9ff6878, Previous frame's sp is 0xe9ff6880
 Saved registers:
  ecx at 0xe9ff6878, edx at 0xe9ff6874, ebp at 0xe9ff6870, eip at 0xe9ff687c
End of Source crash report

How to repeat:
Call mysql_real_connect from MySQL Connector/C 5.7.27 (Generally Available version)

Suggested fix:
I don't know
[1 Feb 2020 12:27] Adrián G.
Description:
Hi, the following crash seems to happen sometimes at mysql_real_connect.

Call is performed inside code from a custom MySQL C wrapper interface library that's loaded dynamically (dlsym/dlopen) by a game server main library. It may be important to note that it's done on a separate thread,

It seems it's caused by malloc. What potential causes could incur into it crashing with a SIGABRT signal?

Thanks

----------------------------------------------
CRASH: Sat Feb  1 12:16:33 CET 2020
Start Line: ./srcds_linux -game hl2mp -debug -dumplongticks +maxplayers 32 +mp_teamplay 1 +map rp_c18_v1
[New LWP 7089]
[New LWP 7075]
[New LWP 7076]
[New LWP 7077]
[New LWP 7078]
[New LWP 7079]
[New LWP 7080]
[New LWP 7081]
[New LWP 7082]
[New LWP 7086]
[New LWP 7087]
[New LWP 7088]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./srcds_linux -game hl2mp -debug -dumplongticks +maxplayers 32 +mp_teamplay 1 +'.
Program terminated with signal SIGABRT, Aborted.
#0  0xf7f9d939 in __kernel_vsyscall ()
[Current thread is 1 (Thread 0xe9ff7b40 (LWP 7089))]
#0  0xf7f9d939 in __kernel_vsyscall ()
#1  0xf7be3832 in raise () from /lib/i386-linux-gnu/libc.so.6
#2  0xf7be4cc1 in abort () from /lib/i386-linux-gnu/libc.so.6
#3  0xf7c2c66b in ?? () from /lib/i386-linux-gnu/libc.so.6
#4  0xf7c2fc70 in ?? () from /lib/i386-linux-gnu/libc.so.6
#5  0xf7c30e38 in malloc () from /lib/i386-linux-gnu/libc.so.6
#6  0xe9b23fc3 in my_raw_malloc (my_flags=16, size=<optimized out>) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/mysys/my_malloc.c:191
#7  my_malloc (key=0, size=396, flags=16) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/mysys/my_malloc.c:54
#8  0xe9b2e7c0 in mysql_socket_vio_new (mysql_socket=..., type=VIO_TYPE_TCPIP, flags=2) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/vio/vio.c:274
#9  0xe9b2eaa3 in vio_new (sd=27, type=VIO_TYPE_TCPIP, flags=2) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/vio/vio.c:293
#10 0xe9b04477 in mysql_real_connect (mysql=0xb0bcd0c, host=0xb6d6a90 "remotemysql.com", user=0xb6d64d0 "<my_username>", passwd=0xb6d63b0 "<my_password>", db=0xb6d6280 "fSkSY53EP5", port=3306, unix_socket=0x0, client_flag=0) at /export/home/pb2/build/sb_0-34537258-1560178776.88/mysql-5.7.27/sql-common/client.c:4277
#11 0xe9ace98d in CMySQLConnection::Connect (this=0xb0bcd00, pHostName=0xb6d6a90 "remotemysql.com", pUserName=0xb6d64d0 "<my_username>", pPassword=0xb6d63b0 "<my_password>", pSchemaName=0xb6d6280 "fSkSY53EP5", port=3306) at /home/vsts/work/1/s/mp/src/game/server/hl2rp/dal/mysql_driver/mysql_driver.cpp:344
#12 0xf1e0b4c4 in CDAL::LoadDatabaseConfiguration (this=0xf2667840 <gDAL>) at /home/vsts/work/1/s/mp/src/game/server/hl2rp/dal/dal.cpp:320
#13 0xf1e0aec9 in CDAL::Run (this=0xf2667840 <gDAL>) at /home/vsts/work/1/s/mp/src/game/server/hl2rp/dal/dal.cpp:208
#14 0xf7b82690 in CThread::ThreadProc(void*) () from bin/libtier0_srv.so
#15 0xf7d983bd in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#16 0xf7caee16 in clone () from /lib/i386-linux-gnu/libc.so.6
No symbol table info available.
eax            0x0      0
ecx            0xe9ff688c       -369137524
edx            0x0      0
ebx            0x2      2
esp            0xe9ff6870       0xe9ff6870
ebp            0xe9ff688c       0xe9ff688c
esi            0x8      8
edi            0x0      0
eip            0xf7f9d939       0xf7f9d939 <__kernel_vsyscall+9>
eflags         0x286    [ PF SF IF ]
cs             0x23     35
ss             0x2b     43
ds             0x2b     43
es             0x2b     43
fs             0x3      3
gs             0x63     99
k0             0x0      0
k1             0x0      0
k2             0x0      0
k3             0x0      0
k4             0x0      0
k5             0x0      0
k6             0x0      0
k7             0x0      0
From        To          Syms Read   Shared Object Library
0xf7e92590  0xf7f528f5  Yes (*)     /lib/i386-linux-gnu/libm.so.6
0xf7e84ad0  0xf7e85a84  Yes (*)     /lib/i386-linux-gnu/libdl.so.2
0xf7df7914  0xf7e3dc78  Yes         bin/libstdc++.so.6
0xf7d969d0  0xf7da620f  Yes (*)     /lib/i386-linux-gnu/libpthread.so.0
0xf7bce610  0xf7d1b386  Yes (*)     /lib/i386-linux-gnu/libc.so.6
0xf7f9eab0  0xf7fb97fb  Yes (*)     /lib/ld-linux.so.2
0xf7ba1e00  0xf7bb10fc  Yes         bin/libgcc_s.so.1
0xf7b7b1d0  0xf7b8e44c  Yes (*)     bin/libtier0_srv.so
0xf7b68910  0xf7b6c3a4  Yes (*)     /lib/i386-linux-gnu/librt.so.1
0xf7a6eec0  0xf7a9d944  Yes (*)     bin/libvstdlib_srv.so
0xf7435460  0xf75a6a10  Yes (*)     bin/dedicated_srv.so
0xf73bc4e0  0xf73cad3c  Yes (*)     bin/libsteam_api.so
0xf7290e50  0xf729cd44  Yes (*)     /lib32/libtinfo.so.5
0xf676ddd0  0xf692d208  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/engine_srv.so
0xf6287d30  0xf62ad3b4  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/soundemittersystem_srv.so
0xf613b1c0  0xf620e324  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/materialsystem_srv.so
0xf3c84cf0  0xf3d2f608  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/studiorender_srv.so
0xf3ae4590  0xf3c03860  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/vphysics_srv.so
0xf3a47a20  0xf3a92ee8  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/datacache_srv.so
0xf3a0d190  0xf3a21cbc  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/shaderapiempty_srv.so
0xf1831f20  0xf20e44e4  Yes         /home/steam/.steam/steamcmd/hl2rp_server/hl2mp/bin/server_srv.so
0xf38a0630  0xf38b5a3c  Yes (*)     /home/steam/.steam/steamcmd/hl2rp_server/bin/scenefilecache.so
0xec2cc880  0xed4d2fb4  Yes (*)     /home/steam/.steam/sdk32/steamclient.so
0xf3844bb0  0xf384b554  Yes (*)     /lib/i386-linux-gnu/libnss_files.so.2
0xf3a01b70  0xf3a04ad4  Yes (*)     /lib/i386-linux-gnu/libnss_dns.so.2
0xf382d660  0xf38394c4  Yes (*)     /lib/i386-linux-gnu/libresolv.so.2
0xeb079440  0xebc547c0  Yes (*)     bin/steamclient.so
0xeaa74170  0xeaa9c0a4  Yes (*)     bin/crashhandler.so
0xe9acd920  0xe9ba1088  Yes         /home/steam/.steam/steamcmd/hl2rp_server/bin/mysql_driver.so
(*): Shared library is missing debugging information.
Dump of assembler code for function __kernel_vsyscall:
   0xf7f9d930 <+0>:     push   %ecx
   0xf7f9d931 <+1>:     push   %edx
   0xf7f9d932 <+2>:     push   %ebp
   0xf7f9d933 <+3>:     mov    %esp,%ebp
   0xf7f9d935 <+5>:     sysenter
   0xf7f9d937 <+7>:     int    $0x80
=> 0xf7f9d939 <+9>:     pop    %ebp
   0xf7f9d93a <+10>:    pop    %edx
   0xf7f9d93b <+11>:    pop    %ecx
   0xf7f9d93c <+12>:    ret
End of assembler dump.
Stack level 0, frame at 0xe9ff6880:
 eip = 0xf7f9d939 in __kernel_vsyscall; saved eip = 0xf7be3832
 called by frame at 0xe9ff69b0
 Arglist at 0xe9ff6878, args:
 Locals at 0xe9ff6878, Previous frame's sp is 0xe9ff6880
 Saved registers:
  ecx at 0xe9ff6878, edx at 0xe9ff6874, ebp at 0xe9ff6870, eip at 0xe9ff687c
End of Source crash report
[1 Feb 2020 12:52] Adrián G.
Update trailing dash typo in OS distro
[2 Feb 2020 18:56] MySQL Verification Team
Thank you for the bug report. Please provide C file test case which allow to repeat the issue?. Attach it using the Files tab.
[8 Feb 2020 13:02] Adrián G.
Hello again,

It seems I fixed my issue. Apparently, it was due to the multithreading management. Since this was also causing an unrelated massive gameplay stuttering issue (which required me to add a small periodic sleep), and given that the architecture which involves the MySQL driver wrapper is quite complex, I prefered to concentrate to fix my issues and not attempt to make any test case which would probably be useless without an undeterministic multithreaded context.

One of the possible causes regarding my original issue, though, is that I had set my separate thread to make every MySQL operations except mysql_close(), which was done on the main thread when I wanted to. Now everything is done in the I/O thread.

Then, I have a final question. Given that my custom dynamically loaded MySQL wrapper library is dlclose()'d along the mysql_close() call in an appropiate destructor, do I also have to call mysql_library_end() right after it at that destructor? Or all resources including possible TCP/IP sockets would be closed automatically along the effective module close?

Thanks in advance.
[10 Feb 2020 13:19] MySQL Verification Team
Hi Mr. G.

mysql_close() is enough, for every mysql MYSQL pointer that you are using ......
[10 Feb 2020 13:50] Adrián G.
Hi Sinisa,

Thanks for answering. I decided to call mysql_library_end() still when module is requested to close via dlclose(), for convenience.

So I pressume bug wasn't mysql_client library fault.

Best regards.
[10 Feb 2020 14:46] MySQL Verification Team
Best regards from me, Mr. G. !!!!!!