MySQL Bugs: #32814: Mono xsp2 web server randomly hangs when it uses Connector.Net

Bug #32814	Mono xsp2 web server randomly hangs when it uses Connector.Net
Submitted:	28 Nov 2007 13:49	Modified:	21 Jul 8:47
Reporter:	Dzugaru Darkness
Status:	Open
Category:	Connector/Net	Severity:	S1 (Critical)
Version:	5.0.8.1, 5.1.3	OS:	Linux (suse 10.3)
Assigned to:	Tonci Grgin	Target Version:
Tags:	mono xsp mod_mono

Description:
I've got this bug recently when I implemented a class, that queries MySql database in a
loop (every 5 seconds). Class is instantiated in Global.asax, and launches one thread
(not using thread pool), that sleeps 5 secs, then updates information from the database,
caching it to in-memory object to speed up responses to users. After random time interval
(it may be 1 hour or 5-6) xsp2 (mono web server) just hangs. It stops serving web
requests, even for pictures! However it continues to accept TCP connections, but they are
just timeout, with no response given. Xsp2 process stops consuming processor time, value
of  memory used freezes, etc. 

I've written simple web app to test it, and i'm sure there are no deadlocks or something
in my code. Only Global.asax and one permanent thread that opens a connection, does
ExecuteScalar() on SELECT, closes connection and sleeps some time, then it all repeats
from beginning. After a hundred (thousand?) iteration all just hangs inside MySql
Connector code (i've placed a log tracers to make sure that's not in my code, but i'll
try to put tracers inside Connector's source too). When I use connection pooling, or
reuse existing connection things get better - it works more time before hanging. But that
time varies greatly.
I can't imagine what can completely hang a managed application! I can't be even sure
that's a Connector trouble, but logging stops inside it.

Tested on 5.1.3 connector, and replacing xsp2 with mod_mono for apache - still there...

Googling gives some links:
http://www.mono-project.com/ThreadPool_DeadLocks (but MySqlConnector doesn't use
ThreadPool does it?)

http://www.nabble.com/mod_mono-failing-t4450525.html (mod_mono freezes reason unknown)

How to repeat:
Create simple database with 1 table(i have 3 fields - index tinyint(4), name
varchar(255), user int(11) unsigned, no indexes)

Create simple web application. Create a class in App_Code folder, my code is:

public class StatusChecker 
{  
 const string connectionString = "server=localhost;user id=lalala; password=wasdwasd;
database=bugTest; pooling=false"; //disable pooling to get bug sooner
    public StatusChecker()
    {
        Thread thread = new Thread(new ThreadStart(Check));
        thread.Start();        
    }
    void Check()
    {
        do
        {            
            Thread.Sleep(100); //lesser you set here sooner will bug occur, but 0 can
prevent webserver from processing requests
            try
            {
                 MySqlConnection c = new MySqlConnection(connectionString);            
            c.Open();     // << it can hang inside here
                  if (c == null || c.State != ConnectionState.Open) continue;        
MySqlCommand command = new MySqlCommand("SELECT `user` FROM `test`", c);
         
            object o = command.ExecuteScalar();// << or even here O_o
(according to log)
            c.Close();             
            }
            catch (Exception e)
            {
                  //write to log    
            }
        } while (true);
    }        
}

Instantiate it in Global.asax to get it "static"

void Application_Start(object sender, EventArgs e) 
{
  StatusChecker statusChecker = new StatusChecker();
  Application["StatusChecker"] = statusChecker;
}

Host an application using xsp2 or mod_mono for apache. 
Load Default.aspx in browser to start it.
Wait for some time (or better launch 10 copies on different ports)
Webserver stops responding.

Suggested fix:
No ideas ( Maybe write a script that checks a website and restarts it if it hangs? But
that's ugly solution (

Guess I found where Connector.Net may use ThreadPool (conflicting with xsp web server
thread pool as described here - http://www.mono-project.com/ThreadPool_DeadLocks)

It's a place in code where executing command can timeout:
 // start a timeout timer
                if (connection.driver.Version.isAtLeast(5, 0, 0) &&
                     commandTimeout > 0)
                {
                    TimerCallback timerDelegate =
                         new TimerCallback(TimeoutExpired);
                    timer = new Timer(timerDelegate, this, this.CommandTimeout * 1000,
Timeout.Infinite);
                }

So it uses System.Threading.Timer, but I don't know if Timer uses ThreadPool... But it
hangs somewhere in ExecuteReader method, that's for sure.

As I thought, all hangs here:

public new MySqlDataReader ExecuteReader(CommandBehavior behavior)
{
   ....
   finally
   {
       canCancel = false;
       if (timer != null)
       {
             timer.Dispose(); // <<<< ---- here!
       }
   }
}

This is some kind of deadlock between xsp and Connector, that involves threading...

Hi Dzugaru and thanks for your report. Please except my apologies for not responding for
way too long.

I do not think this is a problem in MySQL SW but rather in FW and/or WEB server. There
are numerous reports like this one in BugsDB and on internet so I made my own test FW for
such cases spawning multiple threads and doing all sorts of "horrible" things and I can't
repeat it on properly configured WEB server.

Maybe, in future, we'll have enough info to deal with this but, for now, there's nothing
much we can do...