Bug #7868 Danish extra characters not read
Submitted: 13 Jan 2005 12:02 Modified: 16 Mar 2005 8:43
Reporter: Dave Roden Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Query Browser Severity:S3 (Non-critical)
Version:1.1.5 OS:Windows (Windows XP SP2)
Assigned to: Mike Lischke CPU Architecture:Any

[13 Jan 2005 12:02] Dave Roden
Description:
If an SQL script text file containing the Danish extra letters (å ø æ) in data feilds is opened using File->Open Script, the Danish characters are ignored and do not appear in the ultant script window.

Subsequent execution results in any Danish characters not being present during script run, and in the case of INSERTS for example, not being entered into the data base.

How to repeat:
Create text.sql file with the following example commands: 

drop table if exists test_chars ;
create table test_chars ( test_text text );
insert into test_chars (test_text) values ('Danish ae - æ, oe - ø, aa - å' );

Then from MySQL query browser:

File -> Open Script...

select test.sql

Resultant script tab window contains:

drop table if exists test_chars;
create table test_chars ( test_text text );
insert into test_chars (test_text) values ('Danish ae - , oe - , aa - ' );

Note Danish characters are missing.

Exection of the script results in a row in test_text without Danish characters - 'Danish ae - , oe - , aa - '
[13 Jan 2005 14:48] MySQL Verification Team
I can't reproduce it using copy/paste. Please, upload file test.sql for testing.
[13 Jan 2005 19:37] Dave Roden
That was very prompt action Victoria! 

I've uploaded the test_chars.sql file I used in the report. Hope it's helpful.

If there are problems with the file I'll try and think of another way of getting a file with the Danish extra characters to you. 

Dave
[21 Jan 2005 8:54] P Ericsson
It seems like the same thing happens for swedish characters åäö. As far as i can see it goes wrong wen you save a script including these characters. 

And it isn't only for script files this happends. It is also when you save a resultset including åäö.

Regards

Pär
[15 Feb 2005 12:48] Mike Lischke
Are you aware that QB can open scripts in various encodings? By default it uses UTF-8 for loading as well as storing scripts. However the given test.sql script is straight ANSI and opening it with the default character encoding setting leads to wrong interpretation of the content. Set "SQL Script File ANSI (*.sql) in the file type combo box of the open dialog and you will see all characters as expected.

Let me add an advice: In a global world like ours using ANSI or similar code page based encodings is no longer a good move. It is too simple to misinterpret the data, as these formats do not contain any info what the data actually is. UTF-8 does only marginally increase the size of an average script (if at all) and provides exact locale info, so the content can never be misread. So I strongly recommend that you always use Unicode encoding for your scripts (UTF-16 is also ok, but requires more storage). If you need more info about the various Unicode Transformation Formats then you can read this: http://www.unicode.org/faq/utf_bom.html.
[16 Mar 2005 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[16 Mar 2005 7:28] Dave Roden
I tried the suggestion regarding ANSI/UTF-8 and that appears to fix it. I have never used UTF-8 (knowingly) before or read anything about it until now and I suggest that many users would be in a similar position - perhaps the default file type should be ANSI rather than UTF-8 in the drop down.
[16 Mar 2005 8:43] Mike Lischke
It would not be a good decision to make the old and error prone encoding the default. It would solve the issue you had but would produce new ones.