MySQL Bugs: #61094: Connector throws nullreference exception on transaction timeout

Bug #61094	Connector throws nullreference exception on transaction timeout
Submitted:	9 May 2011 6:42	Modified:	3 Jun 2011 8:35
Reporter:	Christos Pavlides	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	Connector / NET	Severity:	S1 (Critical)
Version:	6.4.0	OS:	Any
Assigned to:	Assigned Account	CPU Architecture:	Any
Tags:	NullReference TransactionScope

Description:
During a transaction if the call times out, the connection instead of throwing a timeout exception it throws a null reference exception which causes the application to crash.

How to repeat:
Lock some tables from one thread and start a transaction which uses those tables from another thread. At some point the transaction will timeout and the null reference exception will be thrown.

During the crash on the live servers Windows generates a WER file which contains the following data:
Sig[0].Name=Problem Signature 01
Sig[0].Value=imsp server service.exe
Sig[1].Name=Problem Signature 02
Sig[1].Value=1.0.0.0
Sig[2].Name=Problem Signature 03
Sig[2].Value=4dc60985
Sig[3].Name=Problem Signature 04
Sig[3].Value=MySql.Data
Sig[4].Name=Problem Signature 05
Sig[4].Value=6.4.0.0
Sig[5].Name=Problem Signature 06
Sig[5].Value=4d10c461
Sig[6].Name=Problem Signature 07
Sig[6].Value=4e3
Sig[7].Name=Problem Signature 08
Sig[7].Value=7

Using Ilasm on my binaries, I can see that 4e3 points to the property getter for property Connection (get_connection).

I have read bug #59346 and it seems similar to the issue I am having but my exception is quite different:

 at MySql.Data.MySqlClient.NativeDriver.ExecutePacket(MySqlPacket packetToExecute)
   at MySql.Data.MySqlClient.NativeDriver.SendQuery(MySqlPacket queryPacket)
   at MySql.Data.MySqlClient.Driver.SendQuery(MySqlPacket p)
   at MySql.Data.MySqlClient.Statement.ExecuteNext()
   at MySql.Data.MySqlClient.PreparableStatement.ExecuteNext()
   at MySql.Data.MySqlClient.Statement.Execute()
   at MySql.Data.MySqlClient.PreparableStatement.Execute()
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader()
   at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
   at MySql.Data.MySqlClient.MySqlTransaction.Rollback()
   at MySql.Data.MySqlClient.MySqlTransactionScope.Rollback(SinglePhaseEnlistment singlePhaseEnlistment)
   at MySql.Data.MySqlClient.MySqlPromotableTransaction.System.Transactions.IPromotableSinglePhaseNotification.Rollback(SinglePhaseEnlistment singlePhaseEnlistment)
   at System.Transactions.DurableEnlistmentAborting.EnterState(InternalEnlistment enlistment)
   at System.Transactions.DurableEnlistmentActive.InternalAborted(InternalEnlistment enlistment)
   at System.Transactions.TransactionStateAborted.EnterState(InternalTransaction tx)
   at System.Transactions.TransactionStateActive.Rollback(InternalTransaction tx, Exception e)
   at System.Transactions.EnlistableStates.Timeout(InternalTransaction tx)
   at System.Transactions.Bucket.TimeoutTransactions()
   at System.Transactions.BucketSet.TimeoutTransactions()
   at System.Transactions.TransactionTable.ThreadTimer(Object state)
   at System.Threading._TimerCallback.TimerCallback_Context(Object state)
   at System.Threading.ExecutionContext.runTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading._TimerCallback.PerformTimerCallback(Object state)

Suggested fix:
I did not have time to further debug this but I will try to do this now and see if I can provide more information.

After debugging the actual code to reproduce the problem, I found that the exception occurs at the following function:
private void ExecutePacket(MySqlPacket packetToExecute)
{
  try
  {
      warnings = 0;
      stream.SequenceByte = 0;
      stream.SendPacket(packetToExecute);
  }
  catch (MySqlException ex)
  {
      HandleException(ex);
      throw;
  }
}
The stream is null.

This happens after the driver throws a timeout exception in:
public int GetResult(ref int affectedRow, ref int insertedId)
        {
            try
            {
                packet = stream.ReadPacket();
            }
            catch (TimeoutException)
            {
                // Do not reset serverStatus, allow to reenter, e.g when
                // ResultSet is closed.
                throw;
            }

I believe that the problem lies in the fact that after the TimeoutException the code tries to send a "ROLLBACK" command but the stream/connection was already closed due to the timeout exception.

This also explains why the exception is unhandled, and causes the application to crash. The driver properly throws the timeout exception and the application catches it properly but then a new thread tries to handle the rollback and it is this thread that throws the null exception which cannot be handled by the application.

Guys, do we have any news on this?

Please provide a small database script and Visual Studio project that will quickly reproduce this issue.

The issue is easily reproducable, if you start one thread and lock a table/record and then on a separate thread you start a transaction and try to update the same table/record. The driver instead of timing out it will produce the problem defined here.

Please do not submit the same bug more than once. An existing bug report already describes this very problem. Even if you feel that your issue is somewhat different, the resolution is likely
to be the same. Because of this, we hope you add your comments to the original bug instead.

Duplicate of Bug #59346
Fixed in 6.2.5, 6.3.7 and 6.4.1+.

Thank you for your interest in MySQL.

Dear Julio, thank you for the update, but as you can see from my initial submition I have read issue number 59346 and it does not seem to be the same exception. Also there is no 6.4.0 version in the repository. The latest trunk has a version of 6.4.1. 
Again I would like to thank you for your time and patience. 
I have a feeling though that the last changes you have submitted related to not reusing a closed connection (bug #58316) might have an impact on this issue as well. I will retest and see if the issue is resolved.

I have made a mistake with the versions, this is the corrected comment:

Dear Julio, thank you for the update, but as you can see from my initial submition I have read issue number 59346 and it does not seem to be the same exception. Also there is no 6.4.1 version in the repository. The latest trunk has a version of 6.4.0. 
Again I would like to thank you for your time and patience. 
I have a feeling though that the last changes you have submitted related to not reusing a closed connection (bug #58316) might have an impact on this issue as well. I will retest and see if the issue is resolved.

This bug is still not fixed and is not a duplicate of bug #59346.

http://stackoverflow.com/questions/29094122/c-how-to-catch-exception-in-mysql-rollback

We have run into this issue as well and we are currently using 6.9.3. Definitely not a duplicate of the other bugs mentioned.

Same problem in version 6.9.7

I've asked the developer involved in the other bug to take a look since this might be either not a duplicate or an incomplete fix to the earlier bug.

James Day, MySQL Senior Principal Support Engineer, Oracle

We are experiencing this problem on 6.9.9

This bug is still present in the latest package from nuget as at time of writing: 8.0.11

It's easy to reproduce with code like the following NUnit test:

[Test]
public void Bug_WhenCommandTimesOut_ScopeDisposalThrowsNullReferenceException()
{
  using (var scope = new TransactionScope(
    TransactionScopeOption.Required, 
    new TransactionScopeOptions() 
    { 
      IsolationLevel = IsolationLevel.ReadCommitted,
      Timeout = TimeSpan.FromSeconds(30) 
    }))
  {
    using (var connection = new MySqlConnection("SERVER=localhost, DATABASE=mysql, UID=root, password=root"))
    {
      using (var cmd = connection.CreateCommand())
      {
          cmd.CommandText = "SELECT * from INFORMATION_SCHEMA.TABLES LIMIT 1; SELECT SLEEP(5);";
          cmd.CommandTimeout = 1;
          try
          {
            cmd.ExecuteNonQuery();
          }
          catch (Exception)
          {
            /* ignore: the timeout causes exception here */
          }
      }    
    }
    // scope would be disposed at the end of the block,
    //  of course, but let's be explicit for the bug:
    scope.Dispose(); // FAIL: NullReferenceException.
  }
}

This is a pretty big deal -- it appears to be fixed in 6.9.12 (latest stable 6.9.x from nuget), but is still present in 8.0.11 (latest stable from nuget)

This is still broken in 8.0.16. The offending code can be found in NativeDriver.cs, line 511 according to the decompiled source I have:

```
506    private void ExecutePacket(MySqlPacket packetToExecute)
507    {
508      try
509      {
510        this.warnings = 0;
511        this.stream.SequenceByte = (byte) 0;
512        this.stream.SendPacket(packetToExecute);
513      }
514      catch (MySqlException ex)
515      {
516        this.HandleException(ex);
517        throw;
518      }
519    }
```

And, really, it shouldn't be hard to check that `stream` is not null before working with it. It may even be valid to just do:
```
this.stream?.SequenceByte = (byte) 0;
this.stream?.SendPacket(packetToExecute);
```
Since, if the stream is missing, we're kinda lost already? Or perhaps throw something more meaningful?

This bug is so frequently hit on our systems that I'm literally writing the second implementation of code that does:
1. Is this a NullReferenceException?
2. Does it have MySql.Data.MySqlClient.NativeDriver.ExecutePacket as the first method in the stack
3. Translate this issue into an exception which we can explicitly catch: a TransactionTimedOutException.

It's crap, because I have to assume that there's no other way the stream can end up null there, but I'm left with little choice. I can't just catch all NullReferenceExceptions because it will make _real_ issues a mission to debug. I can't just let this go, because TransactionTimedOut is something we might want to deal with (retry? log? whatever).

This issue has been open since 2011, and, whilst it was apparently fixed (I didn't see this with 6.9.12, but did with every other 6.9 release I tried), it has regressed. Please fix.