Bug #14144 mysql_server_end does not kill the manager thread in sql_manager.cpp
Submitted: 19 Oct 2005 15:28 Modified: 23 Jul 2007 20:45
Reporter: Patrick SUBLAURIER Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Embedded Library ( libmysqld ) Severity:S3 (Non-critical)
Version:4.1.20 OS:Windows (Windows)
Assigned to: Iggy Galarza CPU Architecture:Any

[19 Oct 2005 15:28] Patrick SUBLAURIER
Description:
Hi,

I used the MySQL Server 4.1 Embedded library with Windows 2000/XP.

If you load libmysqld.dll and call successively:
mysql_server_init()
mysql_init()
...
mysql_close()
mysql_server_end() and then unload libmysql.dll, the process will be terminated after a certain delay...

How to repeat:
In fact, when mysql_init is called and flush-time is not null, MySQL will create a manager thread which is defined in the file sql_manager.cpp (pthread_handler_decl).
This thread is waiting for a semaphore with a timeout of FLUSH-TIME (1800 seconds by default).

If you call mysql_close and then mysql_server_end and unload libmysqld.dll the thread is still asleep, waiting for the semaphore... But if the semaphore is released, or the timeout has expired, the thread will execute code in the unloaded dll and will raise an access violation in ntdll.dll which causes the program to crash and terminate without error message or dialog box (unless you debug it !!!)

Suggested fix:
Obviously, you can set flush-time to zero in the file MY.INI...

OR

the mysql_server_end() function must verify if the manager thread exists and kill it.
[20 Oct 2005 8:41] Vasily Kishkin
I was able to reproduce the bug when libmysqld is loaded from program. If libmysqld is linked statically the code work fine. I think it is problem of Delphi, but it is not problem of libmysqld. By the way you can try to build libmysqld without TLS. Could you reproduce the bug on C program ? Because I was not able.
[20 Oct 2005 11:28] Patrick SUBLAURIER
Yes, I could reproduce the bug with a C++ program (see attached files MySQLTest2.zip in Visual Studio). It is not Delphi's fault in fact.

Actually, the problem is that resources that are allocated by libmysqld.dll (semaphore + thread) are not released when mysql_server_end() and FreeLibrary(libmysqld) are called. That has for consequences that the thread is still alive (waiting for the semaphore to be released or abandonned after flush-time timeout).

Everytime you push the button on the sample program (change flush-time value to 30s in sample's my.ini), you will see in the debugger that a new thread will be created (flush table thread in fact)... and in my opinion, that is not normal (memory leak)!

If libmysqld.dll is statically linked to the program, the Access Violation does not appear because the thread's code is still in memory... The resources (semaphores + threads) will only be released by Windows (and not by MySQL) when the process terminates.
[20 Oct 2005 22:20] MySQL Verification Team
Could you please try your sample code changing line 122 of
code to:

FreeLibraryAndExitThread(libmysqld,0);

Thanks in advance.
[21 Oct 2005 8:24] Patrick SUBLAURIER
With FreeLibraryAndExitThread at the line 122, the sample program crashes immediately whathever the value of flush-time !

With FreeLibrary at the line 122, it crashes only if flush-time <> 0 and approximately after "flush-time" seconds.

I send you the 2 versions (MySqlTest3.zip) if you could not compile my sample (Visual Studio 2003)
[21 Oct 2005 8:26] Patrick SUBLAURIER
Copy the 2 programs in the Debug folder contained in of MySQLTest2.zip

Attachment: MySQLTest3.zip (application/x-zip-compressed, text), 75.41 KiB.

[25 Oct 2005 14:05] MySQL Verification Team
Sorry for my delayed reply.

I built your VS sample and ran it against Purify (see partial report at the bottom)
and I was unable to see what you reported. I think would be nice you try
to make a test case with a console application instead of a GUI one for
to reduce the potential leaks outside of the embedded library and then
isolate the issue from GUI stuff. I asked you to try FreeLibraryAndExitThread
because FreeLibrary can unmaps the DLL from the process's address space
immediately and that can be the issue for to release the thread.

[I] Starting Purify'd MySQLTest2.exe at 20/10/2005 19:56:59
        Instrumented executable: C:\Arquivos de programas\Rational\PurifyPlus\cache\MySQLTest2$Purify_C_14144_MySQLTest2_Debug.exe
        Working directory:       C:\14144\MySQLTest2\Debug
        Command line arguments:  <none>
        Process ID:              0xaf8
        Thread ID: 0xa3c
[I] Starting main
[E] BSR: Beyond stack read in chkstk {1 occurrence}
        Reading 4 bytes from 0x0013b318 (top of stack is at 0x0013b514))
        Address 0x0013b318 points into a thread's stack 
        Address 0x0013b318 is 568 bytes above the frame pointer in chkstk
        Thread ID: 0xa3c
        Error location
            chkstk         [memset.asm:79]
            _crtGetStringTypeA [memset.asm:157]
            setSBUpLow     [memset.asm:362]
            setmbcp_lk     [memset.asm:698]
            setmbcp        [memset.asm]
            _initmbctable  [memset.asm:743]
            cinit          [ioinit.c:218]
            LdrLoadDll     [NTDLL.dll]
            VirtualProtect [kernel32.dll]
            EnableMenuItem [USER32.dll]
[E] BSR: Beyond stack read in chkstk {1 occurrence}
        Reading 4 bytes from 0x0013b2dc (top of stack is at 0x0013b4d8))
        Address 0x0013b2dc points into a thread's stack 
        Address 0x0013b2dc is 596 bytes above the frame pointer in chkstk
        Thread ID: 0xa3c
        Error location
            chkstk         [memset.asm:79]
            _crtLCMapStringA [memset.asm:249]
            setSBUpLow     [memset.asm:367]
            setmbcp_lk     [memset.asm:698]
            setmbcp        [memset.asm]
            _initmbctable  [memset.asm:743]
            cinit          [ioinit.c:218]
            LdrLoadDll     [NTDLL.dll]
            VirtualProtect [kernel32.dll]
            EnableMenuItem [USER32.dll]
[E] BSR: Beyond stack read in chkstk {1 occurrence}
        Reading 4 bytes from 0x0013b0dc (top of stack is at 0x0013b2d8))
        Address 0x0013b0dc points into a thread's stack 
        Address 0x0013b0dc is 1108 bytes above the frame pointer in chkstk
        Thread ID: 0xa3c
        Error location
            chkstk         [memset.asm:79]
            _crtLCMapStringA [memset.asm:306]
            setSBUpLow     [memset.asm:367]
            setmbcp_lk     [memset.asm:698]
            setmbcp        [memset.asm]
            _initmbctable  [memset.asm:743]
            cinit          [ioinit.c:218]
            LdrLoadDll     [NTDLL.dll]
            VirtualProtect [kernel32.dll]
            EnableMenuItem [USER32.dll]
[E] BSR: Beyond stack read in chkstk {1 occurrence}
        Reading 4 bytes from 0x0013b2bc (top of stack is at 0x0013b4b8))
        Address 0x0013b2bc points into a thread's stack 
        Address 0x0013b2bc is 596 bytes above the frame pointer in chkstk
        Thread ID: 0xa3c
        Error location
            chkstk         [memset.asm:79]
            _crtLCMapStringA [memset.asm:249]
            setSBUpLow     [memset.asm:372]
            setmbcp_lk     [memset.asm:698]
            setmbcp        [memset.asm]
            _initmbctable  [memset.asm:743]
            cinit          [ioinit.c:218]
            LdrLoadDll     [NTDLL.dll]
            VirtualProtect [kernel32.dll]
            EnableMenuItem [USER32.dll]
[O] Pure: (EIP: 0x3f03e235): Duplicate insert;  anchorHEAP, key=0x4050000
        Select this item and press F1 for information about this error
        Thread ID: 2620
        Version 2003A.06.10 Early Access build 5101;
        WinNT 5.1 2600 Service Pack 2 Multiprocessor Free
        Call location
            RtlCreateHeap  [NTDLL.dll]
            LdrLoadDll     [NTDLL.dll]
            LoadLibraryExA [kernel32.dll]
            _AfxDispatchCmdMsg(CCmdTarget *,UINT,int,(CCmdTarget::*)(void),...) [cmdtarg.cpp:88]
            CCmdTarget::OnCmdMsg(UINT,int,void *,AFX_CMDHANDLERINFO *) [cmdtarg.cpp:396]
            CDialog::OnCmdMsg(UINT,int,void *,AFX_CMDHANDLERINFO *) [dlgcore.cpp:88]
            CWnd::OnCommand(UINT,long) [wincore.cpp:2549]
            CWnd::OnWndMsg(UINT,UINT,long,long *) [wincore.cpp:1759]
            CWnd::WindowProc(UINT,UINT,long) [wincore.cpp:1745]
            AfxCallWndProc(CWnd *,HWND__ *,UINT,UINT,long) [wincore.cpp:241]
        Memory region from 0x0013BD34 for 716 bytes
        0013BD34:  3F 00 01 00 00 00 00 00  00 00 00 00 00 00 00 00 ?...............
       
        <cut>
  [I] Program terminated at 20/10/2005 19:57:09
[25 Oct 2005 22:40] Patrick SUBLAURIER
What it appears to me is that:
The "manager thread" or "flush thread"  is created when the engine is loaded... This thread is executing code in the file sql_manager.cpp. The thread is being waiting for the semaphore contained in the COND_manager structure.

But when mysql_server_end is called, that thread is still alive, because, pthread_cond_signal(&COND_manager) is probably never called when the server is shutting down... In the mysqld.c file, there is a function called "close_connections" which amongst other things signals the end of the flush thread (abort_loop must be TRUE). I did not found the equivalent in the libmysqld project.

I hoped this could help you. I have made the same sample program in a console mode (MySqlTest4.zip). If you debug it you will be able to view the flush threads that are not being released (set flush-time to 30s and press key a few times) and catch the access violation which raises when the semaphore is released by Windows and the thread is trying to continue the execution in the UNLOADED library...
[11 Jul 2006 12:49] MySQL Verification Team
I was unable to repeat this issue:

C:\MySqlTest4\Debug>MySqlTest4
Press any key to load MySql Server 4.1 Embedded and unload it !

Succeeded...

Press any key to load MySql Server 4.1 Embedded and unload it !

Succeeded...

Press any key to load MySql Server 4.1 Embedded and unload it !

C:\MySqlTest4\Debug>

If you are able for to provide a screenshot of the debugger session
showing the issue you reported, would be helpful for to understand
better what you meant. Thanks in advance.
[11 Jul 2006 15:01] Patrick SUBLAURIER
Hello Miguel, pleased to read you after few months...

I can repeat the bug with the latest version of libmysqld.dll (4.1.20) I just downloaded.
If you run the program MySqlTest4 in VS 2003 you will see the Access Violation which crashes the program and stop it, because the AV raises in ntdll.dll.

I send you the message I catch in AccessViolation.jpg
I hope this will help you.
[11 Jul 2006 15:02] Patrick SUBLAURIER
The message I catch in Visual Studio 2003

Attachment: AccessViolation.JPG (image/pjpeg, text), 14.18 KiB.

[11 Jul 2006 15:16] MySQL Verification Team
Thank you for the feedback. Yes I am getting a crash with current
source, however that looks to me a different issue than the reported,
below my call stack:

+	&THR_LOCK_lock	0x1057d580 _THR_LOCK_lock	_RTL_CRITICAL_SECTION *
+	*tmp	{thr_errno=1244516 suspend={waiting=270129959 semaphore=0x00000000 } mutex={DebugInfo=0x0012fd8c {Type=0 CreatorBackTraceIndex=0 CriticalSection=0x0000001c {DebugInfo=??? LockCount=??? RecursionCount=??? ...} ...} LockCount=1 RecursionCount=1244544 ...} ...}	st_my_thread_var
+	THR_KEY_mysys	{thr_errno=??? suspend={waiting=??? semaphore=??? } mutex={DebugInfo=??? LockCount=??? RecursionCount=??? ...} ...}	st_my_thread_var
	end	0x1019ad65 end	void *
	error	0	char
+	tmp	0x0012fd44 {thr_errno=1244516 suspend={waiting=270129959 semaphore=0x00000000 } mutex={DebugInfo=0x0012fd8c {Type=0 CreatorBackTraceIndex=0 CriticalSection=0x0000001c {DebugInfo=??? LockCount=??? RecursionCount=??? ...} ...} LockCount=1 RecursionCount=1244544 ...} ...}	st_my_thread_var *

Can you verify if you got the same call stack as the above.

Thanks in advance.
[17 Jul 2006 16:48] Patrick SUBLAURIER
You obtain that call stack because you used the debug version of libmysqld... If you use the release version (4.1.20), you will be able to get the AV at the address of 0x101A6FF4 which is the address of the EXCEPTION HANDLER of the _thread_start function (see the file thread.c).

So, I maintain what I say from the beginning, the manager thread which executes the pthread_handler_decl function (cf. sql_manager.cpp) is not released by mysql_server_end, because it is still waiting (WaitForSingleObject called by pthread_cond_timedwait in line 62 of sql_manager.cpp) for the semaphore stored in COND_manager.semaphore...

To correct this bug, you'll have to SIGNAL the semaphore and WAIT for the manager thread to be finished when mysql_server_end is called. Here is the code that is present in mysqld.c in the body of close_connections:
  
  /* kill flush thread */
  (void) pthread_mutex_lock(&LOCK_manager);
  if (manager_thread_in_use)
  {
    DBUG_PRINT("quit",("killing manager thread: %lx",manager_thread));
   (void) pthread_cond_signal(&COND_manager);
  }
  (void) pthread_mutex_unlock(&LOCK_manager);
[22 Jul 2006 2:33] MySQL Verification Team
Thank you for the feedback. I have compiled the release version of the
4.0.21 source server with debugging symbols, the crash I am getting is
at:

Unhandled exception at 0x100da9e8 (libmysqld.dll

100DA9E8, LIBMYSQLD.DLL, my_thread_init, 

Anyway earliest version not presents that exception, reason why
I am changing as verified.
[23 Jul 2007 20:45] Iggy Galarza
I can reproduce this bug in mysql 4.1-embedded on Windows.  I cannot reproduce the behaviour in in mysql 5.1-embedded on Windows.  I believe it was fixed by a change to mysql-5.0 (ChangeSet 1.2294.31.13) which included version 1.577.5.1 of mysqld.cc with the comment "Fixed compiler warnings detected by option -Wshadow and -Wunused".  The fix will not be back ported due to it's size.