MySQL Bugs: #11744: Sporadic Character set 'utf8' is not supported errors

Bug #11744	Sporadic Character set 'utf8' is not supported errors
Submitted:	5 Jul 2005 19:49	Modified:	5 Nov 2007 22:42
Reporter:	Mark Modrall	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	Connector / NET	Severity:	S4 (Feature request)
Version:	1.0.4	OS:	Windows (Windows 2000)
Assigned to:		CPU Architecture:	Any

Description:
I have a program reading and processing log files into a database.  The data from the logs is utf8 encoded.  I added 

;pooling=false;charset=utf8

to my connection string for MySqlConnector/.Net and for the most part it works fine but about 1 in 100 executions, the attempt to connect fails with a

Exception: MySql.Data.MySqlClient.MySqlException
Message: Character set 'utf8' is not supported
Source: MySql.Data
   at MySql.Data.MySqlClient.CharSetMap.GetEncoding(DBVersion version, String CharSetName)
   at MySql.Data.MySqlClient.Driver.Configure(MySqlConnection connection)
   at MySql.Data.MySqlClient.NativeDriver.Configure(MySqlConnection connection)
   at MySql.Data.MySqlClient.MySqlConnection.Open()

error.  Nothing changes in the code or the os or the machine.  It just sporadically blows up like this.

How to repeat:
I can't reproduce on command; it just happens now and again even though it works fine most of the time.

Sorry...I was not able to reproduce the bug. Could you please write version of mysqld and my.ini(cfg) file ? I attached my test case. Could you please run the test case on your computer ?

Test case

Attachment: 11744.zip (application/x-zip-compressed, text), 3.79 KiB.

Sorry I didn't provide enough information.  The mysqld we're running is 4.1.12 running on linux.  The my.cnf from the linux system I can attach.  We have no my.ini for windows.

To provide more information about the circumstances in which I see it, this happens more like 1 in every 3 or 400 occurrances.  The app is multithreaded and fetches log files from remote locations; when the fetch is complete, each thread connects to the db and starts processing the log file.  At any given time there would be 8 threads/connections to the db.

The error seems to occur very early in the process (i.e. around the first attempt to connect to the db).  The error happens on the connection.Open() call.  The connections last only for the duration of processing that specific log file; if there are more log files, the connection is re-opened.  Since the same pool of 8 threads process all of the log files, each thread opens and closes connections 7-8 times.  The error does not re-occur after the first instance.

Given the difficulty in reproducing the bug, I figured it would be hard to find a resolution.

Thanks
_Mark

Could you please attach your my.cfg file ?

mysqld configuration

Attachment: my.cnf (application/octet-stream, text), 8.76 KiB.

There it is...  I wasn't sure whether the error being generated would be from the mysqld side or from the .Net connector side...

Got a new variation on the Open error today.  Seems like there's some kind of collision error when multiple threads try to open connections at the same moment.  Usually, I get the "'utf8' is not supported error, but this time I got that and this:
05.07.12 15:06:16 [ERROR]: 
Exception: MySql.Data.MySqlClient.MySqlException
Message: Character set 'utf8' is not supported
Source: MySql.Data
   at MySql.Data.MySqlClient.CharSetMap.GetEncoding(DBVersion version, String CharSetName)
   at MySql.Data.MySqlClient.Driver.Configure(MySqlConnection connection)
   at MySql.Data.MySqlClient.NativeDriver.Configure(MySqlConnection connection)
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at Lycos.LogParser.Lycos50Parser.Process(String logPath, Object userParam) in d:\logprocessing\lycos50\lycos50.cs:line 165
   at Lycos.LogFetcher.BatchRunnerLogs.BRunnerOnComplete(RunnerInstance c, Object userObject) in d:\logprocessing\batchrunnerlogs.cs:line 61

05.07.12 15:06:16 [ERROR]: 
Exception: System.ArgumentException
Message: Item has already been added.  Key in dictionary: "tis620"  Key being added: "tis620"
Source: mscorlib
   at System.Collections.Hashtable.Insert(Object key, Object nvalue, Boolean add)
   at System.Collections.Hashtable.Add(Object key, Object value)
   at MySql.Data.MySqlClient.CharSetMap.LoadCharsetMap()
   at MySql.Data.MySqlClient.CharSetMap.GetEncoding(DBVersion version, String CharSetName)
   at MySql.Data.MySqlClient.Driver.Configure(MySqlConnection connection)
   at MySql.Data.MySqlClient.NativeDriver.Configure(MySqlConnection connection)
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at Lycos.LogParser.Lycos50Parser.Process(String logPath, Object userParam) in d:\logprocessing\lycos50\lycos50.cs:line 165
   at Lycos.LogFetcher.BatchRunnerLogs.BRunnerOnComplete(RunnerInstance c, Object userObject) in d:\logprocessing\batchrunnerlogs.cs:line 61

I had the same bug (plus the bug #4027). It is all about multithreading and thread-safetiness. It is evident that MySqlConnection.Open is not thread safe.
Just add a LOCK on the Open function and the problem is gone

lock(dummySyncObject)
{
  mySqlConnection.Open()
}

I tried to reproduve the bug on 1.0.6 but I was not able. Could you please test the bug on 1.0.6 or create and attach simple test case ?

Hi Vasily...

Being a timing thing, it was never easy getting it down to a 100% reproducible case.  It would happen when I kicked off something like 8 threads and the first thing they all did was to try to open a db connection with ;charset=utf8 on the end.  It would happen once every few days.  It also seemed to happen only at app startup.

I took Slava's suggestion from the last note and added an object to lock around the Open() call and I haven't seen it since.

-Mark

Thanks Slava for your advice.

We have encountered the same error with 1.0.7, .Net 2.0, Windows Server 2003. It is not possible to deterministically reproduce it, it just happens. However, MySql.Data.MySqlClient.CharSetMap is open to a race condition that could lead to the error.

This class stores the encodings in a hashtable that is initialized on the first call to GetEncoding. First the hashtable is created, and then items are added to it. So if another thread (another connection...) requests an encoding in between, it sees an existing but empty hashtable, hence the error.

Creating a temp. hashtable, filling it, and then only assigning it would fix the issue. We plan to use a patched version of the connector and see if the error still occurs.

The race condition is still present in the new 5.0.0 beta.

I can reproduce this problem in connector v1.0.7 but only on a SMP box.

The severity of this bug should not be "Feature request"!

Anyway, the race condition issue has been fixed in 5.x connectors, so the bug should be closed.

Hi, just wondering if this bug still exists in Connector/Net 1.0.10.1?

believe this is a duplicate of 17106 which is now fixed.