Bug #36279 mysql.exe built with Visual C++ 2005 cannot display japanese character.
Submitted: 23 Apr 2008 9:30 Modified: 8 Dec 2008 16:52
Reporter: Meiji KIMURA Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Compiling Severity:S3 (Non-critical)
Version:5.0.58 OS:Microsoft Windows
Assigned to: Vladislav Vaintroub CPU Architecture:Any
Triage: Triaged: D3 (Medium)

[23 Apr 2008 9:30] Meiji KIMURA
Description:
mysql.exe built from source with Visual Studio 2005 cannot display japanese character correctly under japanese environment.

The similar problem reported on MySQL Forum.

mysql.exe that built from source can't display Chinese
http://forums.mysql.com/read.php?117,204440,204440#msg-204440

How to repeat:
[Procedure of build]
OS: Windows XP SP2 (32bit)
Compiler: Visual C++ 2005 Express Edition (japanese edition) + Windows SDK
MySQL: ver 5.0.58 GPL source distribution

I extracted sources and change directory, execute these,
win\configure.js WITH_HEAP_STORAGE_ENGINE WITH_MYISAM_STORAGE_ENGINE WITH_MYISAMMRG_STORAGE_ENGINE WITH_ARCHIVE_STORAGE_ENGINE WITH_BLACKHOLE_STORAGE_ENGINE WITH_EXAMPLE_STORAGE_ENGINE
WITH_FEDERATED_STORAGE_ENGINE WITH_INNOBASE_STORAGE_ENGINE
MYSQLD_EXE_SUFFIX="-nt" __NT__ 
win\build-vs8.bat

Activated Visual C++ with MySql.sln and set 'Release' as solution.
Ignored 'Warning: C4566' to reduce waning error messages,and save
sql_locale.cc as Unicode(UTF-8 with signature(BOM)) - Codepage 65001,
Then execute build in unit 'solution'.

[Procedure of reproduction]
To simplify the problem, I use mysqld of binary distribution as below,

mysql-enterprise-gpl-noinstall-5.0.58-win32.zip

I used the mysql build from source(e.g. mysql-5.0.58\client\release\mysql.exe) and
connect as below,

mysql.exe -uroot test

and execute these SQLs.

use test
set names cp932;
select 'あ';
select hex('あ');

drop table if exists t1;
create table t1 (c1 char(1) charset cp932);
insert into t1 values ("あ");
select c1 from t1;
select hex(c1) from t1;

An result of these statements as below,

[test]> use test
Database changed
[test]> set names cp932;
Query OK, 0 rows affected (0.00 sec)

[test]> select 'あ';
+----+
| あ |
+----+
| |
+----+
1 row in set (0.00 sec)

[test]> select hex('あ');
+-----------+
| hex('あ') |
+-----------+
| 82A0 |
+-----------+
1 row in set (0.00 sec)

[test]>
[test]> drop table if exists t1;
Query OK, 0 rows affected (0.02 sec)

[test]> create table t1 (c1 char(1) charset cp932);
Query OK, 0 rows affected (0.02 sec)

[test]> insert into t1 values ("あ");
Query OK, 1 row affected (0.00 sec)

[test]> select c1 from t1;
+------+
| c1 |
+------+
| |
+------+
1 row in set (0.00 sec)

[test]> select hex(c1) from t1;
+---------+
| hex(c1) |
+---------+
| 82A0 |
+---------+
1 row in set (0.00 sec)

The data are stored in database correctly, but mysql.exe failed to print them.

Suggested fix:
I found that putc() failed to display japanese character in my environment.
I suppose that a cause of this 'setlocale()' of my_init.c line.308.

 308: setlocale(LC_CTYPE, "");             /* To get right sortorder */

This code become an execute file with Visual C++ 2005 that depend on an environment. In japanese environment, 'Japanese_Japan.932' is set as arg2. It harm to LC_CTYPE related function. If argument is set to "C", an execute file become independable one.

To avoid this problem, ommit this line (assuemed setlocal(LC_ALL, "C") or setlocale(LC_CTYPE, "C"); I built mysql.exe without this line, and compiled mysql.exe. It works well.

mysql> use test
Database changed
mysql> set names cp932;
Query OK, 0 rows affected (0.05 sec)

mysql> select 'あ';
+----+
| あ |
+----+
| あ |
+----+
1 row in set (0.00 sec)

mysql> select hex('あ');
+-----------+
| hex('あ') |
+-----------+
| 82A0      |
+-----------+
1 row in set (0.00 sec)

mysql>
mysql> drop table if exists t1;
Query OK, 0 rows affected (0.08 sec)

mysql> create table t1 (c1 char(1) charset cp932);
Query OK, 0 rows affected (0.05 sec)

mysql> insert into t1 values ("あ");
Query OK, 1 row affected (0.00 sec)

mysql> select c1 from t1;
+------+
| c1   |
+------+
| あ   |
+------+
1 row in set (0.00 sec)

mysql> select hex(c1) from t1;
+---------+
| hex(c1) |
+---------+
| 82A0    |
+---------+
1 row in set (0.00 sec)
[23 Apr 2008 12:12] Vladislav Vaintroub
In my opinion

 308: setlocale(LC_CTYPE, "");             /* To get right sortorder */
is really obsolete.

Database users typically have no concerns about locale on the server, if they need right sort order, they specify collations.
[26 Apr 2008 0:19] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/46058

ChangeSet@1.2573, 2008-04-26 02:19:18+02:00, vvaintroub@wva. +1 -0
  Bug#36279 - mysql built with Visual Studio 2005 does not display 
  japanese characters.
  
  Fix - removed setlocale from my_init.c . In MBCS environments
  it caused unwanted character-to-byte translations in fputc() 
  in client code and wrong output as result.
[10 Nov 2008 20:18] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/58372

2711 Vladislav Vaintroub	2008-11-10
       Bug#36279 - mysql built with Visual Studio 2005 does not display 
      japanese characters.
      
      Fix - removed obsolvete setlocale from my_init.c . In MBCS 
      environments it caused unwanted character-to-byte translations
      in fputc()  in client code and wrong output as result.
[12 Nov 2008 11:45] Bugs System
Pushed into 6.0.9-alpha  (revid:vvaintroub@mysql.com-20081110201804-bi98gcs9avsf58ff) (version source revid:alik@sun.com-20081112114133-hyleaxi6lrx4n2gr) (pib:5)
[15 Nov 2008 1:23] Paul Dubois
Noted in 6.0.9 changelog.

The mysql client, when built with Visual Studio 2005, did not display
Japanese characters.

Setting report to NDI pending push into 5.0.x, 5.1.x.
[2 Dec 2008 13:02] Bugs System
Pushed into 5.0.74  (revid:vvaintroub@mysql.com-20081110201804-bi98gcs9avsf58ff) (version source revid:vvaintroub@mysql.com-20081110201804-bi98gcs9avsf58ff) (pib:5)
[2 Dec 2008 21:15] Paul Dubois
Noted in 5.0.74 changelog.

Setting report to NDI pending push into 5.1.x.
[8 Dec 2008 10:21] Bugs System
Pushed into 5.1.31  (revid:vvaintroub@mysql.com-20081110201804-bi98gcs9avsf58ff) (version source revid:azundris@mysql.com-20081114081134-pswy3m5go7r64m1p) (pib:5)
[8 Dec 2008 16:52] Paul Dubois
Noted in 5.1.31 changelog.
[30 Dec 2008 13:16] Miguel Solorzano
Bug: http://bugs.mysql.com/bug.php?id=41777 has been marked as duplicate of this one.
[19 Jan 2009 11:32] Bugs System
Pushed into 5.1.31-ndb-6.2.17 (revid:tomas.ulin@sun.com-20090119095303-uwwvxiibtr38djii) (version source revid:tomas.ulin@sun.com-20090108105244-8opp3i85jw0uj5ib) (merge vers: 5.1.31-ndb-6.2.17) (pib:6)
[19 Jan 2009 13:09] Bugs System
Pushed into 5.1.31-ndb-6.3.21 (revid:tomas.ulin@sun.com-20090119104956-guxz190n2kh31fxl) (version source revid:tomas.ulin@sun.com-20090119104956-guxz190n2kh31fxl) (merge vers: 5.1.31-ndb-6.3.21) (pib:6)
[19 Jan 2009 16:15] Bugs System
Pushed into 5.1.31-ndb-6.4.1 (revid:tomas.ulin@sun.com-20090119144033-4aylstx5czzz88i5) (version source revid:tomas.ulin@sun.com-20090119144033-4aylstx5czzz88i5) (merge vers: 5.1.31-ndb-6.4.1) (pib:6)