MySQL Bugs: #7868: Danish extra characters not read

Bug #7868	Danish extra characters not read
Submitted:	13 Jan 2005 12:02	Modified:	16 Mar 2005 8:43
Reporter:	Dave Roden	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Query Browser	Severity:	S3 (Non-critical)
Version:	1.1.5	OS:	Microsoft Windows (Windows XP SP2)
Assigned to:	Mike Lischke	CPU Architecture:	Any

Description:
If an SQL script text file containing the Danish extra letters (å ø æ) in data feilds is opened using File->Open Script, the Danish characters are ignored and do not appear in the ultant script window.

Subsequent execution results in any Danish characters not being present during script run, and in the case of INSERTS for example, not being entered into the data base.

How to repeat:
Create text.sql file with the following example commands: 

drop table if exists test_chars ;
create table test_chars ( test_text text );
insert into test_chars (test_text) values ('Danish ae - æ, oe - ø, aa - å' );

Then from MySQL query browser:

File -> Open Script...

select test.sql

Resultant script tab window contains:

drop table if exists test_chars;
create table test_chars ( test_text text );
insert into test_chars (test_text) values ('Danish ae - , oe - , aa - ' );

Note Danish characters are missing.

Exection of the script results in a row in test_text without Danish characters - 'Danish ae - , oe - , aa - '

I can't reproduce it using copy/paste. Please, upload file test.sql for testing.

That was very prompt action Victoria! 

I've uploaded the test_chars.sql file I used in the report. Hope it's helpful.

If there are problems with the file I'll try and think of another way of getting a file with the Danish extra characters to you. 

Dave

It seems like the same thing happens for swedish characters åäö. As far as i can see it goes wrong wen you save a script including these characters. 

And it isn't only for script files this happends. It is also when you save a resultset including åäö.

Regards

Pär

Are you aware that QB can open scripts in various encodings? By default it uses UTF-8 for loading as well as storing scripts. However the given test.sql script is straight ANSI and opening it with the default character encoding setting leads to wrong interpretation of the content. Set "SQL Script File ANSI (*.sql) in the file type combo box of the open dialog and you will see all characters as expected.

Let me add an advice: In a global world like ours using ANSI or similar code page based encodings is no longer a good move. It is too simple to misinterpret the data, as these formats do not contain any info what the data actually is. UTF-8 does only marginally increase the size of an average script (if at all) and provides exact locale info, so the content can never be misread. So I strongly recommend that you always use Unicode encoding for your scripts (UTF-16 is also ok, but requires more storage). If you need more info about the various Unicode Transformation Formats then you can read this: http://www.unicode.org/faq/utf_bom.html.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

I tried the suggestion regarding ANSI/UTF-8 and that appears to fix it. I have never used UTF-8 (knowingly) before or read anything about it until now and I suggest that many users would be in a similar position - perhaps the default file type should be ANSI rather than UTF-8 in the drop down.

It would not be a good decision to make the old and error prone encoding the default. It would solve the issue you had but would produce new ones.