MySQL Bugs: #57671: Null Ref Exception when MySql returns error and using CloseConnection behavior

Bug #57671	Null Ref Exception when MySql returns error and using CloseConnection behavior
Submitted:	22 Oct 2010 19:05	Modified:	22 Oct 2010 20:34
Reporter:	Mark Jacobs	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	Connector / NET	Severity:	S3 (Non-critical)
Version:	6.3.5	OS:	Windows (Windows 7 64bit)
Assigned to:		CPU Architecture:	Any
Tags:	CloseConnection, DataReader, NullReferenceException

Description:
My Environment:
- Windows 7 x64
- MySQL Server 5.1
- MySQLConnector 6.5.3
- NET Framework 4.0
- VS 2010

I have discovered that while using the MySqlDataReader that when Calling ExecuteReader with a command behavior of System.Data.CommandBehavior.CloseConnection a NullReferenceException is raised.

I would not expect this to be the error returned. Upon debugging with the MySqlClient source code I discovered during the clean up operations performed on the connection with MySql.Data.MySqlClient a NullReferenceException is raised completely obfuscating the actual error I need to know about.

As the reader executes and processes the error response from the MySql server the cleanup of the reader and connection an exception is thrown in Connection.Abort() which attempts to writes trace information but errors when trying to retrieve the server thread info. At which point we loose the original error message from MySql server.

STACK TRACE FOR RETURNED EXCEPTION:

      at MySql.Data.MySqlClient.MySqlConnection.get_ServerThread()
   at MySql.Data.MySqlClient.MySqlConnection.Abort()
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
   at MySQLSample.Samples.MissingParameters() in c:\users\mark\documents\visual studio 2010\Projects\MySQLSample\MySQLSample\Samples.cs:line 22
   at MySQLSample.Tests.UnitTest1.TestMethod1() in c:\users\mark\documents\visual studio 2010\Projects\MySQLSample\MySQLSample.Tests\UnitTest1.cs:line 17

How to repeat:
Code snippet below; if you add or remove the command behavior this will demonstrate the issue.

MySqlConnection connection = new MySqlConnection("server=localhost;database=myDb;user id=user;password=xxxx");
            try
            {
                MySqlCommand command = new MySqlCommand("up_get_language", connection);
                try
                {
                    connection.Open();
                    MySqlDataReader dr = command.ExecuteReader(System.Data.CommandBehavior.CloseConnection);
                    try
                    {
                        while (dr.Read())
                        {
                            Console.WriteLine(dr.GetString("locale_code"));
                        }
                    }
                    finally
                    {
                        dr.Close();
                        dr.Dispose();
                    }
                }
                finally
                {
                    command.Dispose();
                }
            }
            finally
            {
                if (connection.State == System.Data.ConnectionState.Open)
                {
                    connection.Close();
                }
                connection.Dispose();
            }

Suggested fix:
Additional code is required to handle the CloseConnection command behavior.

Are you saying that the connection should be closed in the event of an exception being throw and it's not being closed?  Simply using CloseConnection is not causing an exception according to my testing. Please give further details.

@Reggie

I have same issues with same configuration ... I'm getting tons of these errors every day.
I'm using it in a multi-threaded environment like asp.net ... 

It is very hard to reproduce this because of the multi-threading issues. You should be more experienced than us in these things. I would really like to help, but I don't know where. My best guess is to stress test the connector in a multi-threaded way for a long period of time and you will definetly get this error.

I would kindly advise you to put some effort over this nasty bug.

at MySql.Data.MySqlClient.MySqlConnection.get_ServerThread()
   at MySql.Data.MySqlClient.MySqlConnection.HandleTimeoutOrThreadAbort(Exception ex)
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()

stack trace

Attachment: stack_trace.txt (text/plain), 6.91 KiB.

Windows 7 x64
MySQL Server 5.5.9
MySQLConnector 6.3.4.0
NET Framework 4.0
VS 2010

I'll start off by saying this bug has caused me a lot of grief and I'm posting here for everyone who is running into this issue. The application I'm working on uses mysql connector, DBLinq and does some heavy multi-threading. Despite multithreaded nature of my application theres only one thread that is used to write to the database. This thread performs a large number of inserts using DBLinq bulk insert. It's not uncommon to see 5-10 HandleTimeoutOrThreadAbort exceptions thrown over 2 hour period. If you would like reproduce this issue download latest version of DBLinq, create a dummy table and try inserting some 1 to 10 million rows of random junk using DBLinq bulk insert.

Over the past two months I've attempted numerous times to resolve this problem correctly. Various timeouts such as command, connection and receive where adjusted, but none of these approaches ever worked. Yesterday I decided it's time to look deeper... I grabbed the latest source code and after spenting some time making it compile (it doesn't compile with VS2010 Ultimate "out of the box") I had a working compilable solution. Next I started looking for a place in the the source code where .NET connector starts making .NET framework calls. To cut to the chase, I've made a couple changes to the .NET connector that appears to have solved the problem. I haven't spend much time testing my changes for any undesired side effects, just keep that in mind if you will decide to use the fix below.

Below are the changes made and the stack trace for HandleTimeoutOrThreadAbort exception.

.NET connector changes made:

1) MySql.Data.MySqlClient.MySqlStream

a) Type of variable inStream was changed from Stream to BufferedStream
b) Added function ReadFullyBuffered which is essentially a copy of ReadFully with the exception that it's first parameter was changed to BufferedStream
c) Function LoadPacket() that used to call ReadFully was changed to call ReadFullyBuffered

2) MySql.Data.MySqlClient.TimedStream

a) Function int Read(byte[] buffer, int offset, int count)

int retval = baseStream.Read(buffer, offset, count);

was changed to

BufferedStream strm = new BufferedStream(baseStream);
int retval = strm.Read(buffer, offset, count);

Stack trace attached, with calls relevant to my application taken out...

We used to get exception from HandleTimeoutOrThreadAbort() method once a month or so. But since version 6.3.5 or 6.3.6 we started to see it regularly, sometimes a few times a day.

I have finally resolved HandleTimeoutOrThreadAbort error.  My previous post was a move in the wrong direction.  What did it for me was a change in network configuration.  Now 10.*.*.* range is used instead of 3.*.*.*.   

I don't fully understand why it matters.

I can very easily reproduce the bug. The exception happens all the time when timeout occurs. Just execute some command which takes more time than CommandTimeout to execute.