Bug #67665 IIS application pool reset worker process causes website to crash
Submitted: 21 Nov 2012 16:18 Modified: 28 Jun 2013 2:13
Reporter: Jiong Mai Email Updates:
Status: Closed Impact on me:
None 
Category:Connector / NET Severity:S1 (Critical)
Version:6.5.6 OS:Microsoft Windows (Server 2003 x64)
Assigned to: Gabriela Martinez Sanchez CPU Architecture:Any
Tags: IIS, session state, w3wp

[21 Nov 2012 16:18] Jiong Mai
Description:
When the IIS application pool resets the worker processes at a specific time, the mysql session state store will crash the w3wp.exe process and the website request will result in a crash error message. This did not happen in version 6.6.3.

Event Viewer messages:

1)

An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/86257515/Root

Process ID: 2216

Exception: System.InvalidOperationException

Message: Connection must be valid and open to rollback transaction

StackTrace:    at MySql.Data.MySqlClient.MySqlTransaction.Rollback()
   at MySql.Web.SessionState.MySqlSessionStateStore.DeleteTimedOutSessionsWithoutCallback()
   at MySql.Web.SessionState.MySqlSessionStateStore.CleanupOldSessions(Object o)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading._TimerCallback.PerformTimerCallback(Object state)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

2)

EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.3959, P3 45d691cc, P4 mysql.data, P5 6.6.4.0, P6 507eeb75, P7 4d6, P8 97, P9 mysql.data.mysqlclient.mysql, P10 NIL.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

3)

An error occurred in the Web application MFIWEB

IP: 192.168.1.83
Form Data:
SortOrder: 
FormAction: ShowFavorites
SearchButton.x: 71
SearchButton.y: 13
Message
Nested transactions are not supported.

Source
MySql.Data

Target site
Void Throw(System.Exception)

Stack trace
   at MySql.Data.MySqlClient.ExceptionInterceptor.Throw(Exception exception)
   at MySql.Data.MySqlClient.MySqlConnection.BeginTransaction(IsolationLevel iso)
   at MySql.Web.SessionState.MySqlSessionStateStore.GetSessionStoreItem(Boolean lockRecord, HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags)
   at MySql.Web.SessionState.MySqlSessionStateStore.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
   at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
   at System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData)
   at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

ToString()

System.InvalidOperationException: Nested transactions are not supported.
   at MySql.Data.MySqlClient.ExceptionInterceptor.Throw(Exception exception)
   at MySql.Data.MySqlClient.MySqlConnection.BeginTransaction(IsolationLevel iso)
   at MySql.Web.SessionState.MySqlSessionStateStore.GetSessionStoreItem(Boolean lockRecord, HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags)
   at MySql.Web.SessionState.MySqlSessionStateStore.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions)
   at System.Web.SessionState.SessionStateModule.GetSessionStateItem()
   at System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData)
   at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

How to repeat:
Updated MySql.Data.dll file in project, MySql.Web.dll file in project, mysql web.config for new version. I also created a new database for the sessions = sessions_d.

		<sessionState timeout="480" cookieless="false" regenerateExpiredSessionId="true" mode="Custom" customProvider="MySqlSessionProvider">
			<providers>
				<add name="MySqlSessionProvider" type="MySql.Web.SessionState.MySqlSessionStateStore, MySql.Web, Version=6.6.4.0, Culture=neutral, PublicKeyToken=C5687FC88969C44D" connectionStringName="MySqlSessionServices" applicationName="mfiweb_dev" autogenerateschema="true"/>
			</providers>
		</sessionState>

		<add name="MySqlSessionServices" connectionString="datasource=192.168.1.183;Use Compression=true;username=xxxxxx;password=xxxxxxxx;database=sessions_d;default command timeout=3600" providerName="MySql.Data.MySqlClient"/>

Then, set the application pool to recycle the worker process in 5 minutes. Logged into website and did some work. When the reset occurs, mysql session state store errors are logged in event viewer. The next time I browse the website I'm taken to an error page. Then, there are more error logs.
[1 Feb 2013 20:15] Frederic MEYER
Hi,

As a matter of fact we've also experienced that exact error message along with w3p3.exe process crash.

But, what is worse, is that we get that message at every single expired session.

Simply follow the tutorial http://dev.mysql.com/doc/refman/5.5/en/connector-net-tutorials-asp-provider-session-state....,

Web.config connectionString:

    <connectionStrings>
        <add name="SessionTestConnectionString" connectionString="server=localhost;port=3306;user id=root;password=password;database=session_test" providerName="MySql.Data.MySqlClient"/>
    </connectionStrings>

Web.config sessionState:
        <sessionState mode="Custom" cookieless="false" timeout="2" regenerateExpiredSessionId="true" customProvider="MySqlSessionStateProvider">
            <providers>
                <add name="MySqlSessionStateProvider" type="MySql.Web.SessionState.MySqlSessionStateStore, MySql.Web, Version=6.6.4.0, Culture=neutral, PublicKeyToken=c5687fc88969c44d" applicationName="/" description="Session Test" connectionStringName="SessionTestConnectionString" writeExceptionsToEventLog="True" autogenerateschema="True" enableExpireCallback="False"/>
            </providers>
        </sessionState>

Get once the Default.aspx page,
Verify that the session is present in my_aspnet_sessions
Wait 2 minutes or so for the session to expire
Refresh the page in the browser and you get:

[InvalidOperationException: Connection must be valid and open to rollback transaction]
   MySql.Data.MySqlClient.MySqlTransaction.Rollback() +137
   MySql.Web.SessionState.MySqlSessionStateStore.CreateUninitializedItem(HttpContext context, String id, Int32 timeout) +486
   System.Web.SessionState.SessionStateModule.CreateUninitializedSessionState() +47
   System.Web.SessionState.SessionStateModule.GetSessionStateItem() +190
   System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +487
   System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +66
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155

If you refresh once more you get:

[InvalidOperationException: Nested transactions are not supported.]
   MySql.Data.MySqlClient.ExceptionInterceptor.Throw(Exception exception) +227
   MySql.Data.MySqlClient.MySqlConnection.Throw(Exception ex) +18
   MySql.Data.MySqlClient.MySqlConnection.BeginTransaction(IsolationLevel iso) +144
   MySql.Data.MySqlClient.MySqlConnection.BeginTransaction() +9
   MySql.Web.SessionState.MySqlSessionStateStore.GetSessionStoreItem(Boolean lockRecord, HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actionFlags) +328
   MySql.Web.SessionState.MySqlSessionStateStore.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) +31
   System.Web.SessionState.SessionStateModule.GetSessionStateItem() +117
   System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +487
   System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +66
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155

The only way to overcome this without restarting both MySQL Server and IIS is the following: using MySQLWorkbench, kill the connection, then delete the orphaned session in the my_aspnet_sessions table. Hit refresh and it works again.

If you do not kill the server connection first, you might no be able to delete the orphaned session because it's locked, and the client would time-out:

ERROR 1205: Lock wait timeout exceeded; try restarting transaction

SQL Statement:

DELETE FROM `session_test`.`my_aspnet_sessions` WHERE `SessionId`='nobozc2i3p5k1l45ofnec1zq' and`ApplicationId`='1'

This has been tested on Windows 7 x64, .NET 4.0, MySql 5.1.66 / 5.5.29 and Connector.NET 6.6.4. To the contrary of what have been said by the original reporter, we do face that issue also with the previous GA release 6.5.5 Connector.NET.

Doing a little of digging in the MySQL Connector sources, it seems that Transactions have been introduced at some point in time in MySql.Web... so I might need to get a version without those to test.
[2 Feb 2013 9:51] Frederic MEYER
Updated info:  the way not to get trapped into the described error is to wait enough time for the session cleanup timer thread to do its job, which weirdly enough, consistently passes every 20 minutes sharp even though it is set to every 10 minutes.

If you ever happen to refresh the page after the session's expiration but before the session cleanup had the time to remove it, you get the above described error.
[2 Feb 2013 10:19] Frederic MEYER
Well, I might have found where this bug relies.

In SessionProvider.cs, in several methods, the connection object is declared in the try block like this:

    public override void CreateUninitializedItem(System.Web.HttpContext context, string id, int timeout)
    {
      MySqlTransaction mySqlTransaction = null;
      try
      {
        using (MySqlConnection conn = new MySqlConnection(connectionString))
        {
...
        }
      }

If something wrong happens, the catch block tries to rollback the transaction, but the way the connection is declared, it has already been disposed once in the catch block:

      catch (MySqlException e)
      {
        if (mySqlTransaction != null)
        {
          try
          {
            Trace.WriteLine("CreateUninitializedItem: Attempt to rollback");
            mySqlTransaction.Rollback();
          }
          catch (MySqlException ex)
          {
            HandleMySqlException(ex, "CreateUninitializedItem: Rollback Failed");
          }
        }
        HandleMySqlException(e, "CreateUninitializedItem");
      }

I think that the connection object should be declared before the first try block.

This pattern happens several times in the SessionProvider.cs source code, also in the CleanupOldSessions callback, hence the original bug report.
[20 May 2013 17:18] Fernando Bichara
This error is masking the real error. In my case, it worked just registering in the GAC (installing Connect / Net on the application server).

Try to find the actual error to know the reason for the drop rollback. After the GAC had other problems, such as the MySQL case sensitive for table names, for example.

As the deletion of expired sessions is done by another thread, any error generates in w3wp crash.

Hope this helps.

Fernando
[20 May 2013 18:03] Frederic MEYER
Hi,

I'm not sure to understand.

If you are saying that there shouldn't have been a rollback situation in the first place, I agree.

But how is it related to the wrong declaration of the connection object (inside the try block) at several places in the SessionProvider.cs file. This is just plain wrong and should be fixed even though another real error situation triggered the rollback.
[20 May 2013 18:21] Fernando Bichara
Hi Frederic,

It's a bug, no question... 

The problem is that the first error does not appear. Any error that is generated in the deletion of sessions that is displayed is InvalidOperationException.

Identifying the original error you can work on a contingency to use in production to correct this situation.

[]'s
[29 May 2013 1:01] Fernando Gonzalez.Sanchez
Yes, the connection must be out of the try block.

Also the reason sessions expired were causing errors was that this line 

byte[] rawSessionItems = (byte[])reader.GetValue(1);

at method SessionProvider.DeleteTimedOutSessionsWithCallback was not prepared to deal with DbNull values.
[4 Jun 2013 16:32] Andrew Frieze
I recently upgraded my .net connector from 6.4.4 to 6.6.5 and now see this error.
[4 Jun 2013 22:37] Gabriela Martinez Sanchez
A patch for this bug has been pushed to the following branches: 6.5.x, 6.6.x, 6.7.x To be released on : 6.5.7,  6.6.6, 6.7.4
[28 Jun 2013 2:13] Philip Olson
Fixed as of the upcoming Connector/Net 6.5.7, 6.6.6, and 6.7.4 releases, and here's the changelog entry:

When the IIS application pool reset the worker processes at a specific
time, the MySQL session state store would crash the "w3wp.exe" process and
the request resulted in a crash error message. There are no longer ASP.NET
crash yellow pages or bad exceptions. Session expiration is now handled
properly.

Thank you for the bug report.