| Bug #67665 | IIS application pool reset worker process causes website to crash | ||
|---|---|---|---|
| Submitted: | 21 Nov 2012 16:18 | Modified: | 28 Jun 2013 2:13 |
| Reporter: | Jiong Mai | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | Connector / NET | Severity: | S1 (Critical) |
| Version: | 6.5.6 | OS: | Windows (Server 2003 x64) |
| Assigned to: | Gabriela Martinez Sanchez | CPU Architecture: | Any |
| Tags: | IIS, session state, w3wp | ||
[21 Nov 2012 16:18]
Jiong Mai
[1 Feb 2013 20:15]
Frederic MEYER
Hi, As a matter of fact we've also experienced that exact error message along with w3p3.exe process crash. But, what is worse, is that we get that message at every single expired session. Simply follow the tutorial http://dev.mysql.com/doc/refman/5.5/en/connector-net-tutorials-asp-provider-session-state...., Web.config connectionString: <connectionStrings> <add name="SessionTestConnectionString" connectionString="server=localhost;port=3306;user id=root;password=password;database=session_test" providerName="MySql.Data.MySqlClient"/> </connectionStrings> Web.config sessionState: <sessionState mode="Custom" cookieless="false" timeout="2" regenerateExpiredSessionId="true" customProvider="MySqlSessionStateProvider"> <providers> <add name="MySqlSessionStateProvider" type="MySql.Web.SessionState.MySqlSessionStateStore, MySql.Web, Version=6.6.4.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d" applicationName="/" description="Session Test" connectionStringName="SessionTestConnectionString" writeExceptionsToEventLog="True" autogenerateschema="True" enableExpireCallback="False"/> </providers> </sessionState> Get once the Default.aspx page, Verify that the session is present in my_aspnet_sessions Wait 2 minutes or so for the session to expire Refresh the page in the browser and you get: [InvalidOperationException: Connection must be valid and open to rollback transaction] MySql.Data.MySqlClient.MySqlTransaction.Rollback() +137 MySql.Web.SessionState.MySqlSessionStateStore.CreateUninitializedItem(HttpContext context, String id, Int32 timeout) +486 System.Web.SessionState.SessionStateModule.CreateUninitializedSessionState() +47 System.Web.SessionState.SessionStateModule.GetSessionStateItem() +190 System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +487 System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +66 System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155 If you refresh once more you get: [InvalidOperationException: Nested transactions are not supported.] MySql.Data.MySqlClient.ExceptionInterceptor.Throw(Exception exception) +227 MySql.Data.MySqlClient.MySqlConnection.Throw(Exception ex) +18 MySql.Data.MySqlClient.MySqlConnection.BeginTransaction(IsolationLevel iso) +144 MySql.Data.MySqlClient.MySqlConnection.BeginTransaction() +9 MySql.Web.SessionState.MySqlSessionStateStore.GetSessionStoreItem(Boolean lockRecord, HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags) +328 MySql.Web.SessionState.MySqlSessionStateStore.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) +31 System.Web.SessionState.SessionStateModule.GetSessionStateItem() +117 System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +487 System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +66 System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155 The only way to overcome this without restarting both MySQL Server and IIS is the following: using MySQLWorkbench, kill the connection, then delete the orphaned session in the my_aspnet_sessions table. Hit refresh and it works again. If you do not kill the server connection first, you might no be able to delete the orphaned session because it's locked, and the client would time-out: ERROR 1205: Lock wait timeout exceeded; try restarting transaction SQL Statement: DELETE FROM `session_test`.`my_aspnet_sessions` WHERE `SessionId`='nobozc2i3p5k1l45ofnec1zq' and`ApplicationId`='1' This has been tested on Windows 7 x64, .NET 4.0, MySql 5.1.66 / 5.5.29 and Connector.NET 6.6.4. To the contrary of what have been said by the original reporter, we do face that issue also with the previous GA release 6.5.5 Connector.NET. Doing a little of digging in the MySQL Connector sources, it seems that Transactions have been introduced at some point in time in MySql.Web... so I might need to get a version without those to test.
[2 Feb 2013 9:51]
Frederic MEYER
Updated info: the way not to get trapped into the described error is to wait enough time for the session cleanup timer thread to do its job, which weirdly enough, consistently passes every 20 minutes sharp even though it is set to every 10 minutes. If you ever happen to refresh the page after the session's expiration but before the session cleanup had the time to remove it, you get the above described error.
[2 Feb 2013 10:19]
Frederic MEYER
Well, I might have found where this bug relies.
In SessionProvider.cs, in several methods, the connection object is declared in the try block like this:
public override void CreateUninitializedItem(System.Web.HttpContext context, string id, int timeout)
{
MySqlTransaction mySqlTransaction = null;
try
{
using (MySqlConnection conn = new MySqlConnection(connectionString))
{
...
}
}
If something wrong happens, the catch block tries to rollback the transaction, but the way the connection is declared, it has already been disposed once in the catch block:
catch (MySqlException e)
{
if (mySqlTransaction != null)
{
try
{
Trace.WriteLine("CreateUninitializedItem: Attempt to rollback");
mySqlTransaction.Rollback();
}
catch (MySqlException ex)
{
HandleMySqlException(ex, "CreateUninitializedItem: Rollback Failed");
}
}
HandleMySqlException(e, "CreateUninitializedItem");
}
I think that the connection object should be declared before the first try block.
This pattern happens several times in the SessionProvider.cs source code, also in the CleanupOldSessions callback, hence the original bug report.
[20 May 2013 17:18]
Fernando Bichara
This error is masking the real error. In my case, it worked just registering in the GAC (installing Connect / Net on the application server). Try to find the actual error to know the reason for the drop rollback. After the GAC had other problems, such as the MySQL case sensitive for table names, for example. As the deletion of expired sessions is done by another thread, any error generates in w3wp crash. Hope this helps. Fernando
[20 May 2013 18:03]
Frederic MEYER
Hi, I'm not sure to understand. If you are saying that there shouldn't have been a rollback situation in the first place, I agree. But how is it related to the wrong declaration of the connection object (inside the try block) at several places in the SessionProvider.cs file. This is just plain wrong and should be fixed even though another real error situation triggered the rollback.
[20 May 2013 18:21]
Fernando Bichara
Hi Frederic, It's a bug, no question... The problem is that the first error does not appear. Any error that is generated in the deletion of sessions that is displayed is InvalidOperationException. Identifying the original error you can work on a contingency to use in production to correct this situation. []'s
[29 May 2013 1:01]
Fernando Gonzalez.Sanchez
Yes, the connection must be out of the try block. Also the reason sessions expired were causing errors was that this line byte[] rawSessionItems = (byte[])reader.GetValue(1); at method SessionProvider.DeleteTimedOutSessionsWithCallback was not prepared to deal with DbNull values.
[4 Jun 2013 16:32]
Andrew Frieze
I recently upgraded my .net connector from 6.4.4 to 6.6.5 and now see this error.
[4 Jun 2013 22:37]
Gabriela Martinez Sanchez
A patch for this bug has been pushed to the following branches: 6.5.x, 6.6.x, 6.7.x To be released on : 6.5.7, 6.6.6, 6.7.4
[28 Jun 2013 2:13]
Philip Olson
Fixed as of the upcoming Connector/Net 6.5.7, 6.6.6, and 6.7.4 releases, and here's the changelog entry: When the IIS application pool reset the worker processes at a specific time, the MySQL session state store would crash the "w3wp.exe" process and the request resulted in a crash error message. There are no longer ASP.NET crash yellow pages or bad exceptions. Session expiration is now handled properly. Thank you for the bug report.
