Bug #40175 Unable to connect to any of the specified MySQL hosts
Submitted: 20 Oct 2008 13:38 Modified: 13 Feb 2009 16:10
Reporter: Luca Leonardo Scorcia Email Updates:
Status: Can't repeat Impact on me:
None 
Category:Connector / NET Severity:S2 (Serious)
Version:5.2.3 OS:Windows
Assigned to: CPU Architecture:Any

[20 Oct 2008 13:38] Luca Leonardo Scorcia
Description:
A couple of weeks ago we upgraded our production machine (Windows Server 2003, IIS, ASP.net 2.0 website, MySql 5.0.67) from MySql Connector.net 5.1.3 to the 5.2.3 version. The upgrade went smoothly and all was fine, until about a week later, when the site stopped working with the following error all over the place (from the Event Viewer):

Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 10/20/2008 2:35:25 PM 
Event time (UTC): 10/20/2008 12:35:25 PM 
Event ID: 1cb80e2c840e4722807acfc8697dd3e1 
Event sequence: 56918 
Event occurrence: 751 
Event detail code: 0 
 
Application information: 
    Application domain: /LM/W3SVC/1196649082/Root-1-128689659783125000 
    Trust level: Full 
    Application Virtual Path: / 
    Application Path: C:\Inetpub\wwwroot\blahblahblah\ 
    Machine name: INCUWEB84-33-51 
 
Process information: 
    Process ID: 3524 
    Process name: w3wp.exe 
    Account name: NT AUTHORITY\NETWORK SERVICE 
 
Exception information: 
    Exception type: MySqlException 
    Exception message: Unable to connect to any of the specified MySQL hosts. 
 
Request information: 
    Request URL: http://blahblahblah/eventi_giorno.aspx?data=25/10/2008 
    Request path: /eventi_giorno.aspx 
    User host address: 79.13.197.209 
    User:  
    Is authenticated: False 
    Authentication Type:  
    Thread account name: NT AUTHORITY\NETWORK SERVICE 
 
Thread information: 
    Thread ID: 1 
    Thread account name: NT AUTHORITY\NETWORK SERVICE 
    Is impersonating: False 
    Stack trace:    at MySql.Data.MySqlClient.NativeDriver.Open()
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at [...]
   at [...].Page_Load(Object sender, EventArgs e)
   at System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e)
   at System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e)
   at System.Web.UI.Control.OnLoad(EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
 
 
Custom event details: 

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The error was fixed by rebooting the machine; however after another week it stopped working again with the same symptoms. Now I have rolled back to the 5.1.7 version, which I hope won't exhibit the same problem.

Connection pooling is activated and normally the application has about 10-20 connections opened. I did not have time to check the number of connections at the moment of the error, since the production machine had to be fixed immediately. If this problem should happen again with version 5.1.7 of the connector I'll try to gather more information.

How to repeat:
I don't really know how to reproduce it. Some kind of stress testing maybe could accelerate the repetition of the bug. All data access in this application is done like follows:

using (MySqlConnection conn = new MySqlConnection(ConfigurationManager.ConnectionStrings["native"].ConnectionString))
{
	conn.Open();

	MySqlCommand cmd = new MySqlCommand(commandText, conn);

	cmd.Parameters.Add("id_forum", MySqlDbType.UInt64).Value = id_forum;
	cmd.Parameters.Add("id_utente", MySqlDbType.UInt64).Value = id_utente;

	using (MySqlDataReader dr = cmd.ExecuteReader())
	{
		List<forum_multe> items = new List<forum_multe>();
					
		while (dr.Read())
		{
			forum_multe item = new forum_multe();
						
			item.id_multa = Convert.ToUInt64(dr["id_multa"]);
			[...]
						
			items.Add(item);
		}
				
		return items;
	}
}

The behavior when the application breaks is similar to the one that happens when the pool runs out of connections, since "some requests work", i.e. some connections probably get killed because of timeouts or something, and then they can get reopened and work. Later requests will be denied until the next "timeout". But this is just a thought, not supported by any fact.
[22 Oct 2008 7:57] Luca Leonardo Scorcia
Today the server experienced the same problem, with connector version 5.1.7. 
I ran a SHOW FULL PROCESSLIST from a root uid, and to my surprise there were ZERO active connections - as I said, pooling is enabled so I expected at least a handful of connections open. Even in the rare occasion of one of the requests working, its DB connection was immediately closed at the end of the web page processing. So I'm thinking that maybe not all of these get marked as closed and returned to the connection pool? 

After running an iisreset /restart (which forces the ASP.net pipeline to reload), the connector started 10 pooled connections as usual.

Is there anything else that I should test to help you diagnosing the problem?
[24 Oct 2008 11:45] Luca Leonardo Scorcia
This is what SHOW FULL PROCESSLIST shows. The first 5 connections come from a Tomcat server running Openfire, they work great during the outage. The sixth connection is used to run SHOW FULL PROCESSLIST using Sqlyog. The remaining others are from ASP.NET requests that fail to authenticate.

Id User Host DB Command Time State Info

1 openfire localhost:1032 openfire Sleep 8 NULL
2 openfire localhost:1048 openfire Sleep 8 NULL
3 openfire localhost:1049 openfire Sleep 8 NULL
4 openfire localhost:1050 openfire Sleep 8 NULL
5 openfire localhost:1051 openfire Sleep 8 NULL
499880 root localhost:4555 NULL Query 0 NULL show full processlist
502212 unauthenticated user localhost:2242 NULL Connect NULL Reading from net NULL
502213 unauthenticated user localhost:2319 NULL Connect NULL Reading from net NULL
502214 unauthenticated user localhost:2411 NULL Connect NULL Reading from net NULL

As usual, running iisreset /restart fixes the issue for a few days by reestablishing the pooled connections.
[30 Oct 2008 18:48] Joao Silva
I´m also getting those errors after upgrading to version 5.2.3:

Error: Unable to connect to any of the specified MySQL hosts.
Details: System.Exception: Exception of type 'System.Exception' was thrown. at MySql.Data.MySqlClient.NativeDriver.Open()
Source: MySql.Data
Details: at MySql.Data.MySqlClient.NativeDriver.Open() at MySql.Data.MySqlClient.MySqlPool.CheckoutConnection() at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection() at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver() at MySql.Data.MySqlClient.MySqlPool.GetConnection() at MySql.Data.MySqlClient.MySqlConnection.Open() at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) at System.Data.Common.DbDataAdapter.Fill(DataTable[] dataTables, Int32 startRecord, Int32 maxRecords, IDbCommand command, CommandBehavior behavior) at System.Data.Common.DbDataAdapter.Fill(DataTable dataTable) at 

and also

Error: Too many connections
Source: MySql.Data
Details: at MySql.Data.MySqlClient.MySqlStream.OpenPacket() at MySql.Data.MySqlClient.NativeDriver.Open() at MySql.Data.MySqlClient.MySqlConnection.Open() at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) at System.Data.Common.DbDataAdapter.Fill(DataTable[] dataTables, Int32 startRecord, Int32 maxRecords, IDbCommand command, CommandBehavior behavior) at System.Data.Common.DbDataAdapter.Fill(DataTable dataTable) at ...

I can´t give any details about the server´s log because my application is hosted in a third party server and i don´t have access to this information.
[20 Nov 2008 17:57] Andreas Falck
We seem to have the same problem with connector with version 5.2.3, see error below.

SQL: select name from dummytable where column1 = @1 and column2 = @2
Exception: MySql.Data.MySqlClient.MySqlException: Unable to connect to any of the specified MySQL hosts. ---> System.Exception: Exception of type 'System.Exception' was thrown.
   at MySql.Data.MySqlClient.NativeDriver.Open()
   --- End of inner exception stack trace ---
   at MySql.Data.MySqlClient.NativeDriver.Open()
   at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at nu.mrorange.storage.DataHandler.GetMySQLConnection(String connectionString)
   at nu.mrorange.storage.DataHandler.GetDataSet(String sql, Object[] arguments, String dataSetName, String tableName, String connectionString)

Problem appeared when doing a upgrade from connector version 5.0.7 to version 5.2.3.

We have several mysql servers and web sites using the connector but just one of the web sites shows this error. There are several web sites connecting to the same MySQL server as the web server showing the error but only the one web server shows this error. 

The web server showing the problem is dedicated. We have servers with higher load not showing the same problem.

It seems that this error first occurs when we run our nightly MySQL backup (ie very high load), but initially the problem is not very severe. After the backup has finished the server is still responsive. The problem gradually becomes more and more intense and around 12 (lunch next day) the server can hardly connect to MySQL at all until we do an IIS reset.

We've changed the timeout in the connection string (was 5000ms due to some antique bug) to 150ms. Same problem.
We changed to the 5.2.5 connector today, hoping it will resolve the issue. We also disabled some extensions in IIS.

As of now we're looking into IIS/IIS extensions, since we only have the problem on one web server. Out of curiosity, do any of you guys also have extensions on your IIS? If so, what extensions?
[13 Feb 2009 16:10] Tonci Grgin
I am sorry but none produced test case I can actually use to verify this... For me, 5.2.3, aside for known problems, works as expected and I've never seen anything like this.
My best guess is that some firewall or something is blocking connection or that server settings changed thus disabling connection authentication.

Should someone come up with test case for this, I'll be monitoring.
[24 Jun 2009 12:01] Yogesh Chaudhari
I am getting below error with MySQL Connector 5.2.5. Is anybody have any suggestion to resolve this. Thanks for the help in advance.

Event Type:	Warning
Event Source:	ASP.NET 2.0.50727.0
Event Category:	Web Event 
Event ID:	1309
Date:		6/24/2009
Time:		6:39:04 AM
User:		N/A
Computer:	229537-WEB1
Description:
Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 6/24/2009 6:39:04 AM 
Event time (UTC): 6/24/2009 11:39:04 AM 
Event ID: 22bc68faec624cfaa01248d77ea4f142 
Event sequence: 81 
Event occurrence: 6 
Event detail code: 0 
 
Application information: 
    Application domain: /LM/W3SVC/1/ROOT-3-128903163027256421 
    Trust level: Full 
    Application Virtual Path: / 
    Application Path: d:\inetpub\wwwroot\ 
    Machine name: 229537-WEB1 
 
Process information: 
    Process ID: 3104 
    Process name: w3wp.exe 
    Account name: NT AUTHORITY\NETWORK SERVICE 
 
Exception information: 
    Exception type: Exception 
    Exception message: Exception of type 'System.Exception' was thrown. 
 
Request information: 
    Request URL: http://www.rxoffice.us/CaseSearchSS.aspx 
    Request path: /CaseSearchSS.aspx 
    User host address: 61.12.21.162 
    User:  
    Is authenticated: False 
    Authentication Type:  
    Thread account name: NT AUTHORITY\NETWORK SERVICE 
 
Thread information: 
    Thread ID: 6 
    Thread account name: NT AUTHORITY\NETWORK SERVICE 
    Is impersonating: False 
    Stack trace:    at MySql.Data.MySqlClient.NativeDriver.Open()
 
 
Custom event details: 

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.