Description:
Run load tests of web pages written in C# on IIS 6.0 (Windows 2003) we can easily reproduce what appear to be deadlocks in the MySQL .NET connector. We've upgraded to 5.2.3 and these deadlocks still appear (although are different to previous versions of the connector we have tried, 1.0.7 and 1.0.10).
An example CLR stack trace is:
0faceab0 7c8285ec [HelperMethodFrame_1OBJ: 0faceab0] System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, UInt32, Boolean, Boolean)
0faceb5c 793b03fe System.Threading.WaitHandle.WaitOne(Int64, Boolean)
0faceb74 793b0c1b System.Threading.WaitHandle.WaitOne(Int32, Boolean)
0faceb84 0f878a75 MySql.Data.MySqlClient.MySqlPool.GetConnection()
0facebb8 0f87846e MySql.Data.MySqlClient.MySqlConnection.Open()
0facebf8 0f87546e nvm.callcentre.vcm.DB..ctor(System.String)
0facec10 0f87526c nvm.callcentre.vcm.DB.getInstance(System.String)
0facec98 0f8750ab nvm.callcentre.vcm.DB.getInstance()
0faceca4 00fd0d88 nvm.callcentre.vcm.Groups.getAgentStats()
0faced14 00fd0cb9 nvm.callcentre.vcm.WebService.getAgentStats()
0facef94 79e7c74b [CustomGCFrame: 0facef94]
0facef78 79e7c74b [GCFrame: 0facef78]
0facef5c 79e7c74b [GCFrame: 0facef5c]
0facf164 79e7c74b [HelperMethodFrame_1OBJ: 0facf164] System.RuntimeMethodHandle._InvokeMethodFast(System.Object, System.Object[], System.SignatureStruct ByRef, System.Reflection.MethodAttributes, System.RuntimeTypeHandle)
0facf1d4 793a44bd System.RuntimeMethodHandle.InvokeMethodFast(System.Object, System.Object[], System.Signature, System.Reflection.MethodAttributes, System.RuntimeTypeHandle)
0facf220 793a41e5 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, Boolean)
0facf26c 793a40a2 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
0facf28c 65d0212c System.Web.Services.Protocols.LogicalMethodInfo.Invoke(System.Object, System.Object[])
0facf2a8 65d17da0 System.Web.Services.Protocols.WebServiceHandler.Invoke()
0facf2e8 65d183b6 System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()
0facf318 65d190d6 System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(System.Web.HttpContext)
0facf32c 0f2e18c0 Microsoft.Web.Services.ScriptHandlerFactory+HandlerWrapper.ProcessRequest(System.Web.HttpContext)
0facf338 65fe6bfb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
0facf36c 65fe3f51 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)
0facf3ac 65fe7733 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)
0facf3fc 65fccbfe System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)
0facf418 65fd19c5 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)
0facf44c 660167d3 System.Web.RequestQueue.WorkItemCallback(System.Object)
0facf464 79407caa System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(System.Object)
0facf468 793740ab System.Threading.ExecutionContext.runTryCode(System.Object)
0facf890 79e7c74b [HelperMethodFrame_PROTECTOBJ: 0facf890] System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object)
0facf8f8 79373ff7 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0facf910 79373ede System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0facf928 79407e18 System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)
0facf93c 79407d90 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)
0facfacc 79e7c74b [GCFrame: 0facfacc]
0facfc18 79e7c74b [ContextTransitionFrame: 0facfc18]
We have this appearing in a heavily used production environment with the default connection string parameters for pooling (i.e. max pool size=100) and we can recreate this easily in a test environment with a max pool size of 20.
How to repeat:
It appears to happen when we place very heavy load on the web application.
We're going to try to provide a simple test case.