Description:
I recently attempted upgrading our large project from MySQL connector .net 8.0.15 to 8.0.28. Suddenly a large number of our unit tests started failing seemingly at random, not the same ones every time.
I can only replicate this when doing async calls, but it is very easy to reproduce and about 90% of the time I run our unit test sweep at least one test will fail. The unit tests are threaded, but not heavily.
I have noticed that if you decrease the size of the connection pool to 10, it does not seem to happen, however the value 100 I am using is the default, and is less than the MySQL max number of connections which I have at 151.
I am attaching a simple program that will demonstrate the issue - this crashes 100% of the time on my Windows 10 x64 computer, but our tests have failed also under Linux with the same error.
The interesting thing is that the same thing can cause two different exceptions. This one is the most common:
Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Already enlisted
at MySql.Data.MySqlClient.Interceptors.ExceptionInterceptor.Throw(Exception exception)
at MySql.Data.MySqlClient.MySqlConnection.Throw(Exception ex)
at MySql.Data.MySqlClient.MySqlConnection.EnlistTransaction(Transaction transaction)
at MySql.Data.MySqlClient.MySqlConnection.Open()
at MySql.Data.MySqlClient.MySqlConnection.<OpenAsync>b__75_0()
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location where exception was thrown ---
at ConnectorNetBug.Program.QueryManyScoped() in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 57
at ConnectorNetBug.Program.Main(String[] args) in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 27
at ConnectorNetBug.Program.<Main>(String[] args)
However, if you add OldGuids=true (this is anecdotal but my impression is that it is related to this setting, I cannot reproduce otherwise, but I might be wrong) to the connection string, then occasionally there can also be a "Value cannot be null" exception instead:
Unhandled exception. System.ArgumentNullException: Value cannot be null.
at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
at MySql.Data.MySqlClient.MySqlTransactionScope.Rollback(SinglePhaseEnlistment singlePhaseEnlistment)
at MySql.Data.MySqlClient.MySqlPromotableTransaction.System.Transactions.IPromotableSinglePhaseNotification.Rollback(SinglePhaseEnlistment singlePhaseEnlistment)
at System.Transactions.DurableEnlistmentAborting.EnterState(InternalEnlistment enlistment)
at System.Transactions.DurableEnlistmentActive.InternalAborted(InternalEnlistment enlistment)
at System.Transactions.TransactionStateAborted.EnterState(InternalTransaction tx)
at System.Transactions.TransactionStateActive.Rollback(InternalTransaction tx, Exception e)
at System.Transactions.Transaction.Rollback()
at System.Transactions.TransactionScope.InternalDispose()
at System.Transactions.TransactionScope.Dispose()
at ConnectorNetBug.Program.QueryManyScoped() in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 64
at ConnectorNetBug.Program.Main(String[] args) in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 27
at ConnectorNetBug.Program.<Main>(String[] args)
How to repeat:
1. Create a new console app targeting .net core 3.1.
2. Add a reference to MySQL.Data 8.0.28
3. Replace the default code with this. Change the database/table/user/password variables as required (it will drop the table name listed):
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Transactions;
namespace ConnectorNetBug
{
internal class Program
{
const string db = "xxx";
const string uid = "root";
const string pwd = "root";
const string tblName = "tests";
const int poolSize = 100;
static string connectionString = $"Server=127.0.0.1;Port=3306;Uid={uid};Password={pwd};Database={db};SslMode=none;MaximumPoolsize={poolSize};ConnectionReset=true;OldGuids=true;";
static async Task Main(string[] args)
{
await Setup();
List<Task> tasks = new List<Task>();
for(int i = 0; i < 2000; i++)
{
var t = QueryManyScoped();
tasks.Add(t);
}
await Task.WhenAll(tasks);
}
private static async Task Setup()
{
using(var sql = GetConnection())
using(var cmd = sql.CreateCommand())
{
await sql.OpenAsync();
cmd.CommandText = $@"drop table if exists tests;
CREATE TABLE `{db}`.`{tblName}` (
`a` int NOT NULL AUTO_INCREMENT,
`b` varchar(10) NOT NULL,
PRIMARY KEY (`a`)
);
";
await cmd.ExecuteNonQueryAsync();
}
}
private static async Task QueryManyScoped()
{
using(var scope = CreateTransactionScope())
{
for(int i = 0; i < 100; i++)
{
using(var sql = GetConnection())
using(var cmd = sql.CreateCommand())
{
await sql.OpenAsync();
cmd.CommandText = @"insert into tests values(null, 'a')";
await cmd.ExecuteNonQueryAsync();
}
}
//scope.Complete();
}
}
public static MySql.Data.MySqlClient.MySqlConnection GetConnection()
{
var conn = new MySql.Data.MySqlClient.MySqlConnection(connectionString);
return conn;
}
/// <summary>
/// Returns a new transaction scope that supports async/await
///
/// Note that transaction timeouts can be implemented here
/// </summary>
/// <param name="existing"></param>
/// <returns></returns>
public static TransactionScope CreateTransactionScope()
{
return new TransactionScope(TransactionScopeOption.Required, new TransactionOptions(), TransactionScopeAsyncFlowOption.Enabled);
}
}
}
Description: I recently attempted upgrading our large project from MySQL connector .net 8.0.15 to 8.0.28. Suddenly a large number of our unit tests started failing seemingly at random, not the same ones every time. I can only replicate this when doing async calls, but it is very easy to reproduce and about 90% of the time I run our unit test sweep at least one test will fail. The unit tests are threaded, but not heavily. I have noticed that if you decrease the size of the connection pool to 10, it does not seem to happen, however the value 100 I am using is the default, and is less than the MySQL max number of connections which I have at 151. I am attaching a simple program that will demonstrate the issue - this crashes 100% of the time on my Windows 10 x64 computer, but our tests have failed also under Linux with the same error. The interesting thing is that the same thing can cause two different exceptions. This one is the most common: Unhandled exception. MySql.Data.MySqlClient.MySqlException (0x80004005): Already enlisted at MySql.Data.MySqlClient.Interceptors.ExceptionInterceptor.Throw(Exception exception) at MySql.Data.MySqlClient.MySqlConnection.Throw(Exception ex) at MySql.Data.MySqlClient.MySqlConnection.EnlistTransaction(Transaction transaction) at MySql.Data.MySqlClient.MySqlConnection.Open() at MySql.Data.MySqlClient.MySqlConnection.<OpenAsync>b__75_0() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj) at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location where exception was thrown --- at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location where exception was thrown --- at ConnectorNetBug.Program.QueryManyScoped() in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 57 at ConnectorNetBug.Program.Main(String[] args) in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 27 at ConnectorNetBug.Program.<Main>(String[] args) However, if you add OldGuids=true (this is anecdotal but my impression is that it is related to this setting, I cannot reproduce otherwise, but I might be wrong) to the connection string, then occasionally there can also be a "Value cannot be null" exception instead: Unhandled exception. System.ArgumentNullException: Value cannot be null. at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken) at MySql.Data.MySqlClient.MySqlTransactionScope.Rollback(SinglePhaseEnlistment singlePhaseEnlistment) at MySql.Data.MySqlClient.MySqlPromotableTransaction.System.Transactions.IPromotableSinglePhaseNotification.Rollback(SinglePhaseEnlistment singlePhaseEnlistment) at System.Transactions.DurableEnlistmentAborting.EnterState(InternalEnlistment enlistment) at System.Transactions.DurableEnlistmentActive.InternalAborted(InternalEnlistment enlistment) at System.Transactions.TransactionStateAborted.EnterState(InternalTransaction tx) at System.Transactions.TransactionStateActive.Rollback(InternalTransaction tx, Exception e) at System.Transactions.Transaction.Rollback() at System.Transactions.TransactionScope.InternalDispose() at System.Transactions.TransactionScope.Dispose() at ConnectorNetBug.Program.QueryManyScoped() in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 64 at ConnectorNetBug.Program.Main(String[] args) in C:\Code\ConnectorNetBug\ConnectorNetBug\Program.cs:line 27 at ConnectorNetBug.Program.<Main>(String[] args) How to repeat: 1. Create a new console app targeting .net core 3.1. 2. Add a reference to MySQL.Data 8.0.28 3. Replace the default code with this. Change the database/table/user/password variables as required (it will drop the table name listed): using System; using System.Collections.Generic; using System.Threading.Tasks; using System.Transactions; namespace ConnectorNetBug { internal class Program { const string db = "xxx"; const string uid = "root"; const string pwd = "root"; const string tblName = "tests"; const int poolSize = 100; static string connectionString = $"Server=127.0.0.1;Port=3306;Uid={uid};Password={pwd};Database={db};SslMode=none;MaximumPoolsize={poolSize};ConnectionReset=true;OldGuids=true;"; static async Task Main(string[] args) { await Setup(); List<Task> tasks = new List<Task>(); for(int i = 0; i < 2000; i++) { var t = QueryManyScoped(); tasks.Add(t); } await Task.WhenAll(tasks); } private static async Task Setup() { using(var sql = GetConnection()) using(var cmd = sql.CreateCommand()) { await sql.OpenAsync(); cmd.CommandText = $@"drop table if exists tests; CREATE TABLE `{db}`.`{tblName}` ( `a` int NOT NULL AUTO_INCREMENT, `b` varchar(10) NOT NULL, PRIMARY KEY (`a`) ); "; await cmd.ExecuteNonQueryAsync(); } } private static async Task QueryManyScoped() { using(var scope = CreateTransactionScope()) { for(int i = 0; i < 100; i++) { using(var sql = GetConnection()) using(var cmd = sql.CreateCommand()) { await sql.OpenAsync(); cmd.CommandText = @"insert into tests values(null, 'a')"; await cmd.ExecuteNonQueryAsync(); } } //scope.Complete(); } } public static MySql.Data.MySqlClient.MySqlConnection GetConnection() { var conn = new MySql.Data.MySqlClient.MySqlConnection(connectionString); return conn; } /// <summary> /// Returns a new transaction scope that supports async/await /// /// Note that transaction timeouts can be implemented here /// </summary> /// <param name="existing"></param> /// <returns></returns> public static TransactionScope CreateTransactionScope() { return new TransactionScope(TransactionScopeOption.Required, new TransactionOptions(), TransactionScopeAsyncFlowOption.Enabled); } } }