Bug #32814 Mono xsp2 web server randomly hangs when it uses Connector.Net
Submitted: 28 Nov 2007 12:49 Modified: 16 Jul 2010 19:48
Reporter: Dzugaru Darkness Email Updates:
Status: Unsupported Impact on me:
None 
Category:Connector / NET Severity:S1 (Critical)
Version:5.0.8.1, 5.1.3 OS:Linux (suse 10.3)
Assigned to: CPU Architecture:Any
Tags: mono xsp mod_mono

[28 Nov 2007 12:49] Dzugaru Darkness
Description:
I've got this bug recently when I implemented a class, that queries MySql database in a loop (every 5 seconds). Class is instantiated in Global.asax, and launches one thread (not using thread pool), that sleeps 5 secs, then updates information from the database, caching it to in-memory object to speed up responses to users. After random time interval (it may be 1 hour or 5-6) xsp2 (mono web server) just hangs. It stops serving web requests, even for pictures! However it continues to accept TCP connections, but they are just timeout, with no response given. Xsp2 process stops consuming processor time, value of  memory used freezes, etc. 

I've written simple web app to test it, and i'm sure there are no deadlocks or something in my code. Only Global.asax and one permanent thread that opens a connection, does ExecuteScalar() on SELECT, closes connection and sleeps some time, then it all repeats from beginning. After a hundred (thousand?) iteration all just hangs inside MySql Connector code (i've placed a log tracers to make sure that's not in my code, but i'll try to put tracers inside Connector's source too). When I use connection pooling, or reuse existing connection things get better - it works more time before hanging. But that time varies greatly.
I can't imagine what can completely hang a managed application! I can't be even sure that's a Connector trouble, but logging stops inside it.

Tested on 5.1.3 connector, and replacing xsp2 with mod_mono for apache - still there...

Googling gives some links:
http://www.mono-project.com/ThreadPool_DeadLocks (but MySqlConnector doesn't use ThreadPool does it?)

http://www.nabble.com/mod_mono-failing-t4450525.html (mod_mono freezes reason unknown)

How to repeat:
Create simple database with 1 table(i have 3 fields - index tinyint(4), name varchar(255), user int(11) unsigned, no indexes)

Create simple web application. Create a class in App_Code folder, my code is:

public class StatusChecker 
{  
 const string connectionString = "server=localhost;user id=lalala; password=wasdwasd; database=bugTest; pooling=false"; //disable pooling to get bug sooner
    public StatusChecker()
    {
        Thread thread = new Thread(new ThreadStart(Check));
        thread.Start();        
    }
    void Check()
    {
        do
        {            
            Thread.Sleep(100); //lesser you set here sooner will bug occur, but 0 can prevent webserver from processing requests
            try
            {
                 MySqlConnection c = new MySqlConnection(connectionString);            
            c.Open();     // << it can hang inside here
                  if (c == null || c.State != ConnectionState.Open) continue;        
MySqlCommand command = new MySqlCommand("SELECT `user` FROM `test`", c);
         
            object o = command.ExecuteScalar();// << or even here O_o
(according to log)
            c.Close();             
            }
            catch (Exception e)
            {
                  //write to log    
            }
        } while (true);
    }        
}

Instantiate it in Global.asax to get it "static"

void Application_Start(object sender, EventArgs e) 
{
  StatusChecker statusChecker = new StatusChecker();
  Application["StatusChecker"] = statusChecker;
}

Host an application using xsp2 or mod_mono for apache. 
Load Default.aspx in browser to start it.
Wait for some time (or better launch 10 copies on different ports)
Webserver stops responding.

Suggested fix:
No ideas ( Maybe write a script that checks a website and restarts it if it hangs? But that's ugly solution (
[29 Nov 2007 6:10] Dzugaru Darkness
Guess I found where Connector.Net may use ThreadPool (conflicting with xsp web server thread pool as described here - http://www.mono-project.com/ThreadPool_DeadLocks)

It's a place in code where executing command can timeout:
 // start a timeout timer
                if (connection.driver.Version.isAtLeast(5, 0, 0) &&
                     commandTimeout > 0)
                {
                    TimerCallback timerDelegate =
                         new TimerCallback(TimeoutExpired);
                    timer = new Timer(timerDelegate, this, this.CommandTimeout * 1000, Timeout.Infinite);
                }

So it uses System.Threading.Timer, but I don't know if Timer uses ThreadPool... But it hangs somewhere in ExecuteReader method, that's for sure.
[29 Nov 2007 8:12] Dzugaru Darkness
As I thought, all hangs here:

public new MySqlDataReader ExecuteReader(CommandBehavior behavior)
{
   ....
   finally
   {
       canCancel = false;
       if (timer != null)
       {
             timer.Dispose(); // <<<< ---- here!
       }
   }
}

This is some kind of deadlock between xsp and Connector, that involves threading...
[21 Jul 2009 6:47] Tonci Grgin
Hi Dzugaru and thanks for your report. Please except my apologies for not responding for way too long.

I do not think this is a problem in MySQL SW but rather in FW and/or WEB server. There are numerous reports like this one in BugsDB and on internet so I made my own test FW for such cases spawning multiple threads and doing all sorts of "horrible" things and I can't repeat it on properly configured WEB server.

Maybe, in future, we'll have enough info to deal with this but, for now, there's nothing much we can do...
[13 May 2010 21:11] freddy rios
I read that 6.2 or 6.3 did away with the Timer in that code, so imho that shouldn't happen anymore in that scenario.
[16 Jul 2010 19:48] Reggie Burnett
We are closing this as not supported for the following reasons:
1.  Our support staff could not reproduce
2.  We are officially not supporting 5.x any longer
3.  Our latest product no longer uses the timer for timeout so the assumption is that the bug doesn't exist any longer

If the