Bug #106398 Connector.net throws "already enlisted" exceptions in case of heavy threading
Submitted: 7 Feb 17:53 Modified: 9 Jun 18:49
Reporter: Karl Johansson Email Updates:
Status: Closed Impact on me:
None 
Category:Connector / NET Severity:S1 (Critical)
Version:8.0.28 OS:Any
Assigned to: CPU Architecture:Any
Tags: regression

[7 Feb 17:53] Karl Johansson
Description:
I recently attempted upgrading our large project from MySQL connector .net 8.0.15 to 8.0.28. Suddenly a large number of our unit tests started failing seemingly at random, not the same ones every time.

I can only replicate this when doing async calls, but it is very easy to reproduce and about 90% of the time I run our unit test sweep at least one test will fail. The unit tests are threaded, but not heavily.

I have noticed that if you decrease the size of the connection pool to 10, it does not seem to happen, however the value 100 I am using is the default, and is less than the MySQL max number of connections which I have at 151.

I am attaching a simple program that will demonstrate the issue - this crashes 100% of the time on my Windows 10 x64 computer, but our tests have failed also under Linux with the same error.

The interesting thing is that the same thing can cause two different exceptions. This one is the most common:

Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Already enlisted
   at MySql.Data.MySqlClient.Interceptors.ExceptionInterceptor.Throw(Exception exception)
   at MySql.Data.MySqlClient.MySqlConnection.Throw(Exception ex)
   at MySql.Data.MySqlClient.MySqlConnection.EnlistTransaction(Transaction transaction)
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at MySql.Data.MySqlClient.MySqlConnection.<OpenAsync>b__75_0()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location where exception was thrown ---
   at ConnectorNetBug.Program.QueryManyScoped() in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 57
   at ConnectorNetBug.Program.Main(String[] args) in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 27
   at ConnectorNetBug.Program.<Main>(String[] args)

However, if you add OldGuids=true (this is anecdotal but my impression is that it is related to this setting, I cannot reproduce otherwise, but I might be wrong)  to the connection string, then occasionally there can also be a "Value cannot be null" exception instead:

Unhandled exception. System.ArgumentNullException: Value cannot be null.
   at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
   at MySql.Data.MySqlClient.MySqlTransactionScope.Rollback(SinglePhaseEnlistment singlePhaseEnlistment)
   at MySql.Data.MySqlClient.MySqlPromotableTransaction.System.Transactions.IPromotableSinglePhaseNotification.Rollback(SinglePhaseEnlistment singlePhaseEnlistment)
   at System.Transactions.DurableEnlistmentAborting.EnterState(InternalEnlistment enlistment)
   at System.Transactions.DurableEnlistmentActive.InternalAborted(InternalEnlistment enlistment)
   at System.Transactions.TransactionStateAborted.EnterState(InternalTransaction tx)
   at System.Transactions.TransactionStateActive.Rollback(InternalTransaction tx, Exception e)
   at System.Transactions.Transaction.Rollback()
   at System.Transactions.TransactionScope.InternalDispose()
   at System.Transactions.TransactionScope.Dispose()
   at ConnectorNetBug.Program.QueryManyScoped() in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 64
   at ConnectorNetBug.Program.Main(String[] args) in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 27
   at ConnectorNetBug.Program.<Main>(String[] args)

How to repeat:
1. Create a new console app targeting .net core 3.1. 
2. Add a reference to MySQL.Data 8.0.28
3. Replace the default code with this. Change the database/table/user/password variables as required (it will drop the table name listed):

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Transactions;

namespace ConnectorNetBug
{
    internal class Program
    {
        const string db = "xxx";
        const string uid = "root";
        const string pwd = "root";
        const string tblName = "tests";
        const int poolSize = 100;
        static string connectionString = $"Server=127.0.0.1;Port=3306;Uid={uid};Password={pwd};Database={db};SslMode=none;MaximumPoolsize={poolSize};ConnectionReset=true;OldGuids=true;";

        static async Task Main(string[] args)
        {
            await Setup();

            List<Task> tasks = new List<Task>();
            for(int i = 0; i < 2000; i++)
            {
                var t = QueryManyScoped();
                tasks.Add(t);
            }
            await Task.WhenAll(tasks);

        }

        private static async Task Setup()
        {
            using(var sql = GetConnection())
            using(var cmd = sql.CreateCommand())
            {
                await sql.OpenAsync();
                cmd.CommandText = $@"drop table if exists tests; 
                    CREATE TABLE `{db}`.`{tblName}`  (
                      `a` int NOT NULL AUTO_INCREMENT,
                      `b` varchar(10) NOT NULL,
                      PRIMARY KEY (`a`)
                    );
                ";
                await cmd.ExecuteNonQueryAsync();
            }
        }

        private static async Task QueryManyScoped()
        {
            using(var scope = CreateTransactionScope())
            {
                for(int i = 0; i < 100; i++)
                {
                    using(var sql = GetConnection())
                    using(var cmd = sql.CreateCommand())
                    {
                        await sql.OpenAsync();
                        cmd.CommandText = @"insert into tests values(null, 'a')";
                        await cmd.ExecuteNonQueryAsync();
                    }
                }
                
                //scope.Complete();
            }
        }

        public static MySql.Data.MySqlClient.MySqlConnection GetConnection()
        {
            var conn = new MySql.Data.MySqlClient.MySqlConnection(connectionString);
            return conn;
        }

        /// <summary>
        /// Returns a new transaction scope that supports async/await
        /// 
        /// Note that transaction timeouts can be implemented here
        /// </summary>
        /// <param name="existing"></param>
        /// <returns></returns>
        public static TransactionScope CreateTransactionScope()
        {
            return new TransactionScope(TransactionScopeOption.Required, new TransactionOptions(), TransactionScopeAsyncFlowOption.Enabled);
        }
    }
}
[7 Feb 17:54] Karl Johansson
Note that the first error happens regardless of whether scope.Complete() is called or not.
[7 Feb 18:18] Karl Johansson
After some more testing I noticed that this happens only on version 8.0.28 - even downgrading to 8.0.27 resolves it.
[8 Feb 7:06] MySQL Verification Team
Hello Karl Johansson,

Thank you for the report and test case.

regards,
Umesh
[8 Feb 15:59] Bradley Grainger
This may be related to the fix for bug #92465, which relates to OpenAsync and TransactionScope. It isn't marked as closed in this bug database, but the 8.0.28 release notes say it includes a fix: https://github.com/mysql/mysql-connector-net/blob/3f7a5259b9809f22872dfd8660dc1ca9fc2326f3...
[9 Jun 18:49] Daniel Valdez
Posted by developer:
 
The bug was fixed by addressing Bug#106243.