Bug #3982 Japanese character on MySQL client command line get garbled
Submitted: 3 Jun 2004 9:26 Modified: 23 Sep 2005 17:42
Reporter: Shuichi Tamagawa Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Command-line Clients Severity:S3 (Non-critical)
Version:5.0.x, 4.1.x OS:Linux (SuSE Linux 9.2)
Assigned to: Shuichi Tamagawa CPU Architecture:Any

[3 Jun 2004 9:26] Shuichi Tamagawa
Description:
When Japanese character (s)  is(are) 'inserted' before the existing characters on the MySQL client command line, the character get garbled. When 'added', It's OK.

The character set variables of my environment are as follows:

| character_set_server |ujis
| character_set_system |utf8
| character_set_database |ujis
| character_set_client |ujis
| character_set_connection |ujis
| character-sets-dir | /usr/local/mysql-standard-4.1.1-alpha-pc-linux-i686/ 
share/mysql/charsets/

How to repeat:
1)Type the command below
mysql> SELECT CONVERT(_ujis ''
 
After that, insert Japanese character(s) between ''. 
mysql> SELECT CONVERT(_ujis 'X'
--->get garbled

2)Type the command below
mysql> SELECT CONVERT(_ujis '
 
After that, add Japanese character(s) after '. 
mysql> SELECT CONVERT(_ujis 'X
--->OK
 
*replace X with Japanese character(s)
[18 Jun 2004 3:22] Miguel Solorzano
It is possible for you to provide a file with the commands
and the Japanese characters for to make possible to test
this bug report ?

Thanks.
[18 Jun 2004 4:09] Shuichi Tamagawa
Japanese characters sample for test encoded in ujis.

Attachment: ujis.txt (text/plain), 102 bytes.

[18 Jun 2004 4:17] Shuichi Tamagawa
Please find attached the file.
This is not a issue of the specific command,
so you can just type any words on the command line
and insert a character by cut&paste from the file.
[1 Jul 2004 8:03] Miguel Solorzano
Sorry I don't understand what are you are reporting
I got messages error when typing the commands showed.

SELECT CONVERT('abc' USING utf8);

the function CONVERT is used like above.

Thanks.
[1 Jul 2004 9:40] Sergei Golubchik
Miguel, it's a known bug in libedit - it's not a server issue.

Let's wait for libedit upgrade (committed, in code-review now) and test it again.
[9 Sep 2004 15:23] Victor Vagin
Actually there are many client-side stuff for japanese text over libedit..
- cannaserver
- kinput2
- kterm
- japanese fonts
or something alike..

wrong setting for these things could cause the described behaviour 

for instance i've checked working of mysqlclient in shell window of xemacs and everything was ok (--with-readline and --with-libedit..).

please check that all settings are right for you
check how it will work in usual shell mode
if it works fine, please describe your configuration (cannaserver, console, and so on) in detail.
[9 Sep 2004 23:38] Shuichi Tamagawa
shell script to run ./configure

Attachment: config-ver (application/octet-stream, text), 518 bytes.

[9 Sep 2004 23:49] Shuichi Tamagawa
Some additional informations: 

This problem happens only to comand line editing of mysql client.
Japanese input for other applications works totally fine (see the example below).

[terminal]
kterm, mlterm, konsole, gnome-terminal
mysql command line editing -> NG for all the terminals
bshell command line editing -> OK for all the terminals

[editer]
emacs, vi, KWrite -> OK

[others]
Mozilla Firefox, Mozilla Thunderbird -> OK

As you mentioned, I'm using cannaserver for conversion system and kinput2 for input system, but the configuration is the same as default. Could you please tell me what exactly you need about these configuration information? (I'm using default configuration for canna and kinput2)

Here is new finding.
When I use binary installation, the problem is as described in my original report.
When I use source installation, I cannot even input any Japanese at all. (After typing, mysql doesn't even display the character)

For your information I've attached the shell script that I used to run "./configure".
[16 Jun 2005 18:43] Shuichi Tamagawa
Changed the Severity
[11 Jul 2005 10:30] Alexander Barkov
Shuichi, 

Brian suggested to reassign this bug to you.

It is most likely bug in libedit/readline.
I suggest to prove it with a small application,
and write a letter to libedit/readline people.
[12 Jul 2005 1:43] Shuichi Tamagawa
Actually, the problem is with bundled readline library, which is version 4.3.

In version 5.0, which is the latest, many multi-byte related bugs have been fixed (http://cnswww.cns.cwru.edu/~chet/readline/CHANGES). If I have mysql use the system readline library, which is version 5.0 on SuSE linux Pro 9.2, by compiling '--without-readline' option, it works fine.
[12 Jul 2005 1:52] Jim Winstead
i added this to our 5.0 tree as an experiment once -- it patches in pretty cleanly. we should do this for 5.1.
[19 Jul 2005 22:32] Jim Winstead
Bundled readline library upgraded to 5.0, which should fix this problem. (Closing, because that is already documented.)
[20 Jul 2005 2:18] Shuichi Tamagawa
With the latest bk version, I still have the same problem. I'm trying to figure out what else is different when compiled --without-readline option.
[13 Sep 2005 1:42] Jim Winstead
the changes to config_readline.h aren't correct -- there are a number of things here that are system-dependent. we need to add the tests for those to our configure setup so that these get set in the top level config.h.
[14 Sep 2005 1:44] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/29780
[15 Sep 2005 4:47] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/29881
[21 Sep 2005 18:25] Shuichi Tamagawa
Fixed in 5.0.14.
[23 Sep 2005 17:42] Paul Dubois
Noted in 5.0.14 changelog.