MySQL Bugs: #312: utf8 doesn't work under Windows

Bug #312	utf8 doesn't work under Windows
Submitted:	22 Apr 2003 3:42	Modified:	25 Apr 2006 4:56
Reporter:	Alexander Barkov	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S3 (Non-critical)
Version:	4.1.0	OS:	Windows (Windows)
Assigned to:	Alexander Barkov	CPU Architecture:	Any

Description:
UTF8 doesn't work in these cases:

CREATE TABLE  () CHARACTER SET utf8;
CREATE TABLE (a CHAR CHARACTER SET utf8);

It seems utf8 is just not compiled.

How to repeat:
I'm not sure as far as I don't have windows

Suggested fix:
First of all we should check if HAVE_CHARSET_utf8 is defined

The problem occurs only on windows binaries for 4.1.0 alpha

The mysqlclient and libmysql libraries has missed the
below files:

\strings\ctype-utf8.c
\strings\ctype-win1250ch.c
\strings\ctype-big5.c

and is necessary to add the same #define like Unix
for to the VC++ compiler not missed the _utf8 character
set.

I have 4.1.0 alpha Win binaries, running mysqld-max-nt.exe.

>SHOW CHARSET;

results:
'big5','Big5 Traditional Chinese','big5','1'
'dec8','DEC West European','dec8_swedish_ci','1'
'cp850','DOS West European','cp850_general_ci','1'
'hp8','HP West European','hp8_english_ci','1'
'koi8r','KOI8-R Relcom Russian','koi8r_general_ci','1'
'latin1','ISO 8859-1 West European','latin1_swedish_ci','1'
'latin2','ISO 8859-2 Central European','latin2_general_ci','1'
'swe7','7bit Swedish','swe7_swedish_ci','1'
'ascii','US ASCII','ascii_general_ci','1'
'ujis','EUC-JP Japanese','ujis','1'
'sjis','Shift-JIS Japanese','sjis','1'
'cp1251','Windows Cyrillic','cp1251_bulgarian_ci','1'
'hebrew','ISO 8859-8 Hebrew','hebrew','1'
'tis620','TIS620 Thai','tis620','1'
'euckr','EUC-KR Korean','euckr','1'
'koi8u','KOI8-U Ukrainian','koi8u_general_ci','1'
'gb2312','GB2312 Simplified Chinese','gb2312','1'
'greek','ISO 8859-7 Greek','greek','1'
'cp1250','Windows Central European','cp1250_general_ci','1'
'gbk','GBK Simplified Chinese','gbk','1'
'latin5','ISO 8859-9 Turkish','latin5_turkish_ci','1'
'armscii8','ARMSCII-8 Armenian','armscii8_general_ci','1'
'utf8','UTF-8 Unicode','utf8','1'
'ucs2','UCS-2 Unicode','ucs2','1'
'cp866','DOS Russian','cp866_general_ci','1'
'keybcs2','DOS Kamenicky Czech-Slovak','keybcs2','1'
'macce','Mac Central European','macce','1'
'macroman','Mac West European','macroman','1'
'cp852','DOS Central European','cp852_general_ci','1'
'latin7','ISO 8859-13 Baltic','latin7_general_ci','1'
'cp1256','Windows Arabic','cp1256_general_ci','1'
'cp1257','Windows Baltic','cp1257_ci_ai','1'
'binary','Binary pseudo charset','binary','1'

>CREATE DATABASE dbname CHARACTER SET utf8;

results:
[root@MYCOMPUTER:3306] ERROR 1115: Unknown character set: 'utf8'

We have to check all of these then:

SELECT _big5'test'
SELECT _latin2'test' COLLATE latin2_czech_ci
SELECT _ujis'test'
SELECT _sjis'test'
SELECT _tis620'test'
SELECT _euckr'test'
SELECT _gb2312'test'
SELECT _cp1250'test' COLLATE cp1250_czech_ci
SELECT _gbk'test'
SELECT _utf8'test'
SELECT _ucs2 0xAABB
SELECT _binary 'test'

Alexander, if you don't have Windows, I can run some SQL commands on the 4.1.0alpha binaries if that would help.  BStil

I'd like someone to check if these queries are working
under windows. Thanks! 

SELECT _big5'test'
SELECT _latin2'test' COLLATE latin2_czech_ci
SELECT _ujis'test'
SELECT _sjis'test'
SELECT _tis620'test'
SELECT _euckr'test'
SELECT _gb2312'test'
SELECT _cp1250'test' COLLATE cp1250_czech_ci
SELECT _gbk'test'
SELECT _utf8'test'
SELECT _ucs2 0xAABB
SELECT _binary 'test'

Adding the characters define on config-win.h I got only
one linkage problem with:

#define HAVE_CHARSET_tis620 1

the test results are showed below:

c:\mysql\bin>mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2 to server version: 4.1.1-alpha-max-debug

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SELECT _big5'test';
+------+
| test |
+------+
| test |
+------+
1 row in set (0.13 sec)

mysql> SELECT _latin2'test' COLLATE latin2_czech_ci;
+---------------------------------------+
| _latin2'test' COLLATE latin2_czech_ci |
+---------------------------------------+
| test                                  |
+---------------------------------------+
1 row in set (0.06 sec)

mysql> SELECT _ujis'test';
+------+
| test |
+------+
| test |
+------+
1 row in set (0.00 sec)

mysql> SELECT _sjis'test';
+------+
| test |
+------+
| test |
+------+
1 row in set (0.00 sec)

mysql> SELECT _tis620'test';
ERROR 1054: Unknown column '_tis620' in 'field list'
mysql> SELECT _euckr'test';
+------+
| test |
+------+
| test |
+------+
1 row in set (0.00 sec)

mysql> SELECT _gb2312'test';
+------+
| test |
+------+
| test |
+------+
1 row in set (0.00 sec)

mysql> SELECT _cp1250'test' COLLATE cp1250_czech_ci;
+---------------------------------------+
| _cp1250'test' COLLATE cp1250_czech_ci |
+---------------------------------------+
| test                                  |
+---------------------------------------+
1 row in set (0.03 sec)

mysql> SELECT _gbk'test';
+------+
| test |
+------+
| test |
+------+
1 row in set (0.00 sec)

mysql> SELECT _utf8'test';
+------+
| test |
+------+
| test |
+------+
1 row in set (0.00 sec)

mysql> SELECT _ucs2 0xAABB;
+---+
|   |
+---+
| ? |
+---+
1 row in set (0.00 sec)

mysql> SELECT _binary 'test';
+------+
| test |
+------+
| test |
+------+
1 row in set (0.00 sec)

Great! We are getting closer!
Thanks a lot, Miguel!

tis620 doesn't work on Unix too.
I'll check this asap.

I have installed mysql-max-nt and there are too many errors:

mysql> SELECT _big5 'test';
ERROR 1054: Unknown column '_big5' in 'field list'
mysql> SELECT _latin2'test' COLLATE latin2_czech_ci;
ERROR 1251: COLLATION 'latin2_czech_ci' is not valid for CHARACTER SET 'latin2'
mysql> SELECT _ujis'test';
ERROR 1054: Unknown column '_ujis' in 'field list'
mysql> SELECT _sjis 'test';
ERROR 1054: Unknown column '_sjis' in 'field list'
mysql> SELECT _tis620'test'
    -> ;
ERROR 1054: Unknown column '_tis620' in 'field list'
mysql> SELECT _euckr'test';
ERROR 1054: Unknown column '_euckr' in 'field list'
mysql> SELECT _gb2312'test';
ERROR 1054: Unknown column '_gb2312' in 'field list'
mysql> SELECT _cp1250'test' COLLATE cp1250_czech_ci;
ERROR 1251: COLLATION 'cp1250_czech_ci' is not valid for CHARACTER SET 'cp1250'
mysql> SELECT _gbk'test';
ERROR 1054: Unknown column '_gbk' in 'field list'
mysql> SELECT _utf8'test';
ERROR 1054: Unknown column '_utf8' in 'field list'
mysql> SELECT _ucs2 0xAABB;
ERROR 1064: You have an error in your SQL syntax.  Check the manual that corresponds to your MySQL s
erver version for the right syntax to use near '0xAABB' at line 1
mysql> SELECT _binary 'test';
+------+
| test |
+------+
| test |
+------+
1 row in set (0.00 sec)

mysql>

Miroslav your are using the servers shipped with
the release 4.1.0 which have problems.

The test results I sent was compiling a server with
a bk tree cloned yesterday for to resolve the issues
for the next release. If you want to test, you need
the VC++ 6.0 compiler more a recent source from BK.

Miguel Solorzano,

When this bug is fixed, can you tell us how to get the most recent Windows binaries with the fixes?  Or source that we must compile with Visual C++?

Will be fixed in next 4.1.1 release according Mark's comments.

UTF8 support is crucial to a project we're doing, and a rapid fix is ideal.  Since I cannot tell how long till the next version comes out (should be soon, I'd think), could any tell me the steps to truly enable UTF8 support?  I tried adding the character (UTF8) DEFINES to config-win.h (to no apparent avail).

OR is there a place I can download the newest version of the windows build from the thing you guys call a BK tree?

If anyone has gotten UTF8 working then please post here what you modified or email me at tlittle@tgrnet.com.  Of course, never mind if the new version will be out within a couple of days.

As work around, please try:

For the project of libmysql and mysqlclient add the below files:

\strings\ctype-utf8.c
\strings\ctype-win1250ch.c
\strings\ctype-big5.c

Add the below lines on confi-win.h

#define HAVE_CHARSET_utf8
#define HAVE_CHARSET_big5
#define HAVE_CHARSET_latin1_de

re-build all the stuff

Could someone send me the recompiled mysqlclient and libmysql libraries for the Win32 4.1 release that fixes this problem and supports the ut8 character set? Perhaps Miguel Solorzano? Thanks a lot. Have a nice life!

Assigned to Mark to close it when a new release including these fixes has been done.

Could someone please post the windows binaries with utf8 support?  It appears indefinite when the next official binary release will be posted on the main mysql.com site.

half a year later and still no updated binary for windows?

Yes, due to very heavy development of new features and fixing bugs 4.1.1 release was delayed several times.

Our policy is to hold release until all known bugs are fixed.

So, when will be the fix available? Would it be a couple of days? a week? a month? We don't have VC++, can anyone kindly send the compiled fix to me??

Can anybody send to me compiled version for windows with UTF8 support! It is very important!

I forget my mail address:  hattila@energosoft.co.yu

Thank you for your bug report. This issue has already been fixed
in the latest released version of that product, which you can download at 
http://www.mysql.com/downloads/

Fixed in 4.1.1.

I have version mysql-4.1.3b-beta-win.zip and it utf8.xml not present in c:\mysql\share\charsets where the rest are can someone point me to a release where utf-8 works?.

I receive the following error
File 'c:\mysql\share\charsets\?.conf' not found (Errcode: 22)
Character set '#33' is not a compiled character set and is not specified in the 'c:\mysql\share\charsets\Index' file

I have mysql 4.1.4 gamma and Im getting the same problem in windows

mysqld - character set utf 8 is not a compiled character set etc etc

and the file for utf8 is indeed not in the share/charsets directory

how can I get a windows version that has utf8 support?

funny thing is that creating tables with it works, what doesnt work is starting the server with utf8 which is essential to connect php to mysql with a connection that
works with utf8

my email is
jglifedesign@yahoo.it

thank u very much
Javier

Javier, there should not be an UTF8 file in share/charset directory. It is
compiled into mysqld. 

If CREATE TABLE works with UTF8 then everything seems to be fine with UTF8.

Please try to run "mysqld --default-character-set=utf8", or write
default-character-set=utf8 into your my.ini.

I have started a project using utf8 mysql database. I have 4.1.7 version of the server. The problem seems not to be fixed.

Can anyone help anyhow?

Thnx

I have the same error using utf8 on Linux 2.6.8 and MySQL server 4.1.7

File '/usr/share/mysql/charsets/?.conf' not found (Errcode: 2)
Character set '#33' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index' file

/etc/my.cnf contains:
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
default-character-set=utf8
default-collation=utf8_general_ci

The last message is generated by mysqld 4.0.x.
It looks like you're trying to start an old mysld with
a new my.cnf file.

I have 4.1.7 and definitely no 4.0.x files knocking about...

and I get the same #33 message coming up...

Once I take:

default-character-set=utf8  out of my.cnf  then the problem disappears...

Ooops... that is on Redhat F2...

How do you start mysqld? Using mysqld_safe? Or directly?

Can you please check if this works (just for testing purposes):
default-character-set=german1

Thank you.

I have the same problem on Windows,
I`m using MySQL 4.1.8-essential.
Do I have to download the newest version, or can I fix it by myself?

Error is: File 'c:\mysql\share\charsets\?.conf' not found (Errcode: 22) Character set '#33' is not a compiled character set and is not specified in the 'c:\mysql\share\charsets\Index' file

But I installed MySQL in C:\Program Files\MySQL\Mysql 4.1.

I have ver 4.1.8 and am getting the same error on Windows Server 2003 box.  Please help.

I am using MySQL version 5.0.11 alpha and work on window platform.

I am able set character set to utf8 without error prompting but everytime when i open DB, it seen the character set reset to the default instead of utf8. When i enter data in utf8 format, only '?' display instead of the chinese character i want.

I have check the mysql/share/character folder and found that inside contain only readme file. Do I need to copy the utf8.conf file into? if yes where to find the file?

Anyone know why the data change to ????, please advise me=) thanks

I am getting the same error in my log files. Namely:

"File 'c:\mysql\share\charsets\?.conf' not found (Errcode: 22)
Character set '#33' is not a compiled character set and is not specified in the 'c:\mysql\share\charsets\Index' file
"

These errors appear when I have "default-character-set=utf8" in my.ini
If I remove that line, the error goes away but then my website (which has some foreign language content being read from a utf8 DB) doesnt display the characters (I just get the ???? values displayed).

Why is the addition of that line causing these errors? And since I need the line, how do I resolve this?

I have tried MySQL 4.1.9 on Windows XP, and 4.1.10 on Windows 2003 Server, both using PHP 4.3.10 (cpgnuke). The problem occurs on both.

Is there anyway to hide or not display these errors (ie.

File 'c:\mysql\share\charsets\?.conf' not found (Errcode: 2) Character set '#33' is not a compiled character set and is not specified in the 'c:\mysql\share\charsets\Index' file)

I really am not needing the unicode so much as I need the error to not show up and ruin my site.

Thanks.

Does anyone read these Bug comments at mysql.com and work towards the solution?

I, too, am experiencing the same problem:
Character set '#33' is not a compiled character set and is not specified in the 
'C:\mysql\\share\charsets\Index' file File 'C:\mysql\\share\charsets\?.conf' not found (Errcode: 2)

I'm using Apache 2.0.52 that came bundled with MySQL 4.1.7 in XAMPP's phpMyAdmin. My Apache error log file contains about 32,400 error lines, all the same error message, as above. And I'm only using as a "localhost" server at home as my off-line web site test-bed on a Windows XP machine!

My biggest problem is trying to get french characters to display properly in phpBB posts - accented letters won't store in the database and the posts only display ????? in place of accented letters.

A fix would be most appreciated.
Doug Brydges, Ontario, Canada
E-mail: freelanc@sympatico.ca

Have you tried using a new version where this is _fixed_?

MySQL-4.1.7 is quite old, considering that we ship 4.1.11 now.

I am still getting this in my Apache error logs after I upgraded to 4.1.11, as suggested. So, the issue does not seem to have been resolved for Windows platforms.

I installed 4.1.11 and had the same error message.
File installed: mysql-4.1.11-win32.zip
OS: Windows 2003 Standard

Here are steps I configured to MySQL 4.1.11 and get error message:
	Configuration Type: Detail Configuration
	Server Type: Developer Machine (use minimum memory)
	Database Usage: Multifunctional database
	Inno data : C:\Program Files\MySQL\Data Files\
	Number of Concurrent connections to the server: Decision Support (DSS)/OLAP
Enabled TCP/IP Networking: checked, Port number: 3306
efault Character Set: Best Support for Multilingualism
Installed as Windows Service: checked
Launched the MySQL server automatically: checked
Include bin directory into Windows PATH: unchecked
Modified security settings: checked, and, put root password
Enabled root access from remote machine: checked
	Create an anonymous account: unchecked

I have an alpha version above 5.0. The bug is present in it either. The utf8 with Russian letters work upon setting utf8 as a default charset in my.ini file and using this charset in Database fields and in Queries. However, each query sent from PHP under Apache 2.0 is followed with an error mesage written in Apache error.log file:

File 'c:\mysql\share\charsets\?.conf' not found (Errcode: 22) Character set '#33' is not a compiled character set and is not specified in the 'c:\mysql\share\charsets\Index' file

I have the same UTF-8 problem with the latest 4.x versions. Does anyone have a solution yet?

- PHP 4.36
- MySQL 4.1.14
- Windows 2000 Server

Pasi

I have the same issue. Error Message is exactly like others reported for utf8 db, character set #33.

Environment:
OS: FC3
MySql: 4.1.11
PHP:  4.3.10

I have set the default-character-set=utf8 in the mysqld, mysql, & client sections of my.cfg.

Same error on version 4.1.18.

Does anyone actually read this bug reports besides the people who are getting bugged by this error?

MySQL: 4.1.18
PHP: 4.4.2
OS: Windows XP SP2
Install Dir: C:\Program Files\MySQL\MySQL Server 4.1
default-character-set=utf8 

Same error:
File 'c:\mysql\share\charsets\?.conf' not found (Errcode: 22) Character set '#33' is not a compiled character set and is not specified in the 'c:\mysql\share\charsets\Index' file

The source of this problem is in PHP. It's compiled against mysql-4.0 client library,
which does not support utf8. Client library tries to initialize utf8, then fails,
and then switches to the default mode, which is usually latin1.

To fully get read of error messages in error.log, one needs to :
- compile PHP with mysql-4.1 client library. On Unix, use --with-mysql=/path/to/4.1/library.
I.e. don't allow PHP to compile against the bandled version of libmysqlclient.
- or install a later version of PHP, which is compiled against at least 4.1 library.

Please consult PHP site about which PHP version uses which MySQL library.

There are several workarounds, which do not need reinstalling PHP.
First of all you need to decide which character set you really
want on the client side. If you don't need utf8, then just configure server not to use it.
I.e. find things like default-character-set in its startup command line or
my.cnf file and fix to latin1 (or to some other character set, depending on localization).

If you do need utf8, then check what "SHOW VARIABLES LIKE 'character_set%'"
reports for you after connection from a PHP program.
If character_set_client, character_set_results and character_set_connection are set to utf8,
then just ignore the error messages, everything should be working fine.

If they're set not to utf8, then:
- add --skip-character-set-client-handshake into server startup command line or into its my.cnf file and restart it,
- or add "SET NAMES utf8" as the first query after connection.
After that check "SHOW VARIABLES LIKE 'character_set%'" from your PHP
program again. They should report utf8. Note, these two ways won't help
to get read of error messages in Apache's error.log too, just ignore them.

I create multilingual webpages in several european language and I had the same problem with mysql, php under windows iis. (I use XML-XSL technology.)
But the solution is very simple - as i realized after many attempts.
I write the php files (default and included also) in ANSII and the rest (MySql DB, XML, XSL, CSS, JS files) are in UTF-8. It works properly without any error.