MySQL Bugs: #62473: Crash (Access violation in 0x0000000)-multiple ODBC Connections on same thread

Bug #62473	Crash (Access violation in 0x0000000)-multiple ODBC Connections on same thread
Submitted:	19 Sep 2011 13:43	Modified:	28 May 2013 14:12
Reporter:	Andre Lorbach	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Connector / ODBC	Severity:	S1 (Critical)
Version:	5.1.8	OS:	Windows
Assigned to:	Bogdan Degtyariov	CPU Architecture:	Any

Description:
We are using the ODBC MYSQL Driver in a multithreading environment. It is possible that a user configures multiple instances which all their own ODBC ENV, Database and Statement handles. 

We do expirience a crash which leads to the MyODBC driver when the following scenario is set (see "How to repeat" for more). 

The exception address is NULL and the callstack is NULL, there is noting that points to the problem. My guess is that the some functions pointers are set to NULL for the concerning thread, regardless if the thread has other Connections open with this ODBC driver. If necessary I can create a small code sample which reproduces the problem very easily. 

How to repeat:
- Thread 1 opens ODBC "Connection A" and processes a statement using SQLPrepare/SQLExecute. Then a WaitableTimer (APC) is set to close "Connection A" automatically after 10 seconds if inactivity. 
- Thread 1 opens a ODBC "Connection B" and prepares another Statement. During the SQLPrepare call, the threads executes the APC which was created previously. "Connection A" is closed by Thread 1 by SQLFreeStmt, SQLDisconnect, SQLFreeConnect and SQLFreeEnv API. 
- Thread 1 returns from the SQLPrepare Call in "Connection B" and tries SQLExecute. The process crashes with access violation at 0x00000000: 
"Unhandled exception at 0x00000000 in process.exe: 0xC0000005: Access violation at location 0x0000000000000000."

Could you please provide complete test case (code) how to repeat the issue. Thanks.

It is extremly hard to reproduce. I have tried to create a small code sample which does pretty much what I think happens after a lot of analyzing and debugging. However I am not getting it to crash. All I know is when it happens, SQLPrepare seems to be doing something and is making the thread idle. Once the thread becomes IDLE, it executes the APC Timer. SQLPrepare never returns, and the application crashes. 

However in my code sample, I am not getting SQLPrepare to make the thread idle, so an APC can be called. So something must be going on inside the ODBC Driver. I will keep trying to create a small sample.

Sample application to reproduce the crash

Attachment: MysqlODBCCrash.rar (application/x-rar-compressed, text), 5.72 KiB.

Regarding the attachment I uploaded, it contains a sample application which produces a crash simular to which I am expiriencing. 

It requires a SystemDSN called "mysqltest" and a table called "testtable", the create table script is in the comments. It opens Connection A, then Connection B, then prepares and executes Statement A on Connection A. then a APC Timer is created which will close the Connection A once the the timer hits and the Thread becomes idle. If this happens while Statement B is processed on Connection B, the crash occurs within the driver. 

That's where my debugging possibilities are ending ;).

One last thing, it happens only if I build use an x64 build. If I build the 32bit version, the problem does not happen.

Andre, thank you very much for providing the test case and all explanations.
Like you said, the crash only occurs with 64-bit build.

Setting the report status "Verified".

A new version 5.1.9 of Connector/ODBC has been released few days ago. 
Need to try it too.

I have just checked the test case source code and found one potential issue.
The callback function does not receive any arguments, it actually modifies (frees) the global handlers. It could happen that the loop went for the next iteration and started initializing handlers in ConnectODBC() when the callback nullified them...

It is a good idea to enclose the allocation area into the critical section.
Also, if the next loop iteration started without freeing handlers from previous time it will cause the memory and connection leaks.

I tried to make the sample code as small as possible, so all values are hardcoded. Our real code is multithreaded and much more complex.

ConnectODBC waits until the APC is finished, and reaches the point where bool a_bConnected is set to false. The SleepEx in ConnectODBC waits until bConnected is set to false before initialising.

My guess is that the crash happens somewhere between the SQLExecute(b_hStmtS) statements.

My idea about going for another iteration has been indirectly confirmed.
I tried to enable ODBC trace and see which ODBC function was called last. Surprisingly, there was no crash at all. Without the trace it was always under 10 iterations in the loop before the crash. With the trace (which slows down ODBC pretty much) it made over 100000 iterations before I stopped it as pointless.

Looks like the flaw was with the asynchronous deallocating of ENV/DBC/STMT handlers.

I increased the delays (SleepEx(100, true)) to make sure the next iteration does not start before all handlers are properly disposed. The program did lots of loops without crashing.

This is definitely not a bug.

If it is not a bug inside the ODBC Driver, what else is it? And why does the problem not happend with the 32Bit Version of the ODBC Drivers? I can run the same code without problems for hours with the 32Bit Version of the MYSQL ODBC drivers. I also tried MSSQL Express 2005 with 64Bit MSSQL Drivers, no crash. 

When I incremented the Sleep to 100, it just tool a little longer to produce the crash but it happens. 

In our real code, when SQLExecute is called with "Connection 1" it somewhere sets the current used Thread to IDLE, which in some cases causes a APC Timer to hit and close "Connection 2". Once the SQLExecute from "Connection 1" resumes it's work the crash happens! So when the handles for "Connection 2" were freed, something got corrupted for "Connection 1" causing the crash. 

Sure the sample code uses hardcoded values, but only to produce this race condition.

> If it is not a bug inside the ODBC Driver, what else is it? 

It looks like the problem with threads synchronization in your code, which at least should use mutex or windows critical section in the callback function and in the connect part.

> And why does the problem not happend with the 32Bit Version 
> of the ODBC Drivers?

That is a good question because the driver uses the same code 
for 32 and 64 bit versions.

Another good question is why enabling the ODBC Trace prevents crashes.
Can you enable ODBC Trace and see if your program crashes?

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

I'm closing this bug because I can not continue without feedback from the reporter. If you have new info, please reopen the report.