Bug #45012 my_like_range_cp932 generates invalid string

Submitted: 21 May 2009 17:06 Modified: 14 Oct 2010 13:30
Reporter: Sergei Golubchik Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Charsets Severity:S2 (Serious)
Version:4.1, 5.0, 5,1, 6.0 bzr OS:Any
Assigned to: Alexander Barkov CPU Architecture:Any

[21 May 2009 17:06] Sergei Golubchik
Description:
my_like_range_cp932() uses 0xFF as a max_sort_char, but it's not a valid cp932 character. As a result my_like_range_cp932 generates invalid cp932 string, which is then passed down to a storage engine in records_in_range().

In particular, DB2 engine doesn't like invalid strings.
See http://lists.mysql.com/internals/36709

How to repeat:
.
[22 May 2009 8:53] Sveta Smirnova
Thank you for the report.

Verified as described.
[22 May 2009 14:00] Tim Clark
Additionally, the following character sets appear to have a similar problem:
big5
euckr
gb2312
gbk
sjis
tis620
ujis
[22 May 2009 20:19] Tim Clark
I wanted to note that this problem is visible with MyISAM also, via a warning:

>CREATE TABLE t1 (c char(10), v varchar(20), index(c), index(v)) collate big5_chinese_ci engine=myisam;
>insert into t1 values ("abc","def"),("abcd", "def"),("abcde","defg"),("aaaa","bbbb");
>select * from t1 force index(v) where v like "x%";
Empty set, 1 warning (0.02 sec)
>show warnings;
+---------+------+-------------------------------------------------------------------------------+
| Level   | Code | Message                                                                       |
+---------+------+-------------------------------------------------------------------------------+
| Warning | 1366 | Incorrect string value: '\xFF\xFF\xFF\xFF\xFF\xFF...' for column 'v' at row 1 |
+---------+------+-------------------------------------------------------------------------------+
[1 Jun 2009 21:10] Tim Clark
Not sure whether this should be put in a different bug report, but I believe that the max_sort_char (0xFFFF) for the ucs2_* collations is also invalid according to the Unicode specification (max valid is 0xFFFD).
[21 Jul 2010 13:58] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/114042

3497 Alexander Barkov	2010-07-21
      Bug#45012 my_like_range_cp932 generates invalid string
      
      Problem: The functions my_like_range_xxx() returned
      a badly formed maximum strings for Asian character sets,
      which made problems for storage engines.
      
      Fix: 
      - Remove a number my_like_range_xxx() implementations,
        which were in fact dumplicate code.
      - Using generic my_like_range_mb() instead
      - Setting max_sort_char member properly
      
      Notes:
      
      - No additional tests available.
        Old tests covers the affected code well enough.
      
      - Also, it's not possible to demonstrate in tests
        that the problem has really disappeared, because
        my_like_range() results cannot be exposed to SQL level.
        This has been checked manually in gdb:
        my_like_range() now returns well-formed strings for all
        Asian character sets.
[22 Jul 2010 11:47] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/114129

3497 Alexander Barkov	2010-07-22
      Bug#45012 my_like_range_cp932 generates invalid string
      
      Problem: The functions my_like_range_xxx() returned
      badly formed maximum strings for Asian character sets,
      which made problems for storage engines.
      
      Fix: 
      - Removed a number my_like_range_xxx() implementations,
        which were in fact dumplicate code pieces.
      - Using generic my_like_range_mb() instead.
      - Setting max_sort_char member properly for Asian character sets
      - Adding unittest/strings/strings-t.c, 
        to test that my_like_range_xxx() return well-formed 
        min and max strings.
      
      Notes:
      
      - No additional tests in mysql/t/ available.
        Old tests cover the affected code well enough.
[26 Jul 2010 5:18] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/114311

3456 Alexander Barkov	2010-07-26
      Bug#45012 my_like_range_cp932 generates invalid string
      
      Problem: The functions my_like_range_xxx() returned
      badly formed maximum strings for Asian character sets,
      which made problems for storage engines.
      
      Fix: 
      - Removed a number my_like_range_xxx() implementations,
        which were in fact dumplicate code pieces.
      - Using generic my_like_range_mb() instead.
      - Setting max_sort_char member properly for Asian character sets
      - Adding unittest/strings/strings-t.c, 
        to test that my_like_range_xxx() return well-formed 
        min and max strings.
      
      Notes:
      
      - No additional tests in mysql/t/ available.
        Old tests cover the affected code well enough.
[26 Jul 2010 7:02] Alexander Barkov
Pushed into mysql-5.1-bugteam (5.1.50)
Pushed into mysql-trunk-merge (5.5.6)
[29 Jul 2010 6:26] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/114583

3460 Alexander Barkov	2010-07-29
      Postfix for BUG#45012.
      
      Problem: The original patch didn't compile on debug_werror
      due to wrong format in printf("%d") for size_t variables.
      
      Fix: Adding cast to (int).
[3 Aug 2010 17:45] Paul DuBois
Noted in 5.1.50 changelog.

The my_like_range_xxx() functions returned badly formed maximum
strings for Asian character sets, which caused problems for storage
engines. 

Setting report to Need Merge pending further pushes.
[4 Aug 2010 7:52] Bugs System
Pushed into mysql-trunk 5.5.6-m3 (revid:alik@sun.com-20100731131027-1n61gseejyxsqk5d) (version source revid:alik@sun.com-20100731074942-o840woifuqioxxe4) (merge vers: 5.5.6-m3) (pib:18)
[4 Aug 2010 8:08] Bugs System
Pushed into mysql-trunk 5.6.1-m4 (revid:alik@ibmvm-20100804080001-bny5271e65xo34ig) (version source revid:alik@sun.com-20100731075120-qz9z8c25zum2wgmm) (merge vers: 5.6.99-m4) (pib:18)
[4 Aug 2010 8:24] Bugs System
Pushed into mysql-trunk 5.6.1-m4 (revid:alik@ibmvm-20100804081533-c1d3rbipo9e8rt1s) (version source revid:alik@sun.com-20100731075120-qz9z8c25zum2wgmm) (merge vers: 5.6.99-m4) (pib:18)
[4 Aug 2010 9:03] Bugs System
Pushed into mysql-next-mr (revid:alik@ibmvm-20100804081630-ntapn8bf9pko9vj3) (version source revid:alik@sun.com-20100731075120-qz9z8c25zum2wgmm) (pib:20)
[4 Aug 2010 22:39] Paul DuBois
Noted in 5.5.6 changelog.
For 5.6, bug does not appear in any released 5.6.x version.
[19 Aug 2010 15:41] Bugs System
Pushed into mysql-5.1 5.1.51 (revid:build@mysql.com-20100819151858-muaaor6jojb5ouzj) (version source revid:build@mysql.com-20100819151858-muaaor6jojb5ouzj) (merge vers: 5.1.51) (pib:20)
[28 Sep 2010 11:53] Alexander Barkov
See also Bug#50677
[14 Oct 2010 8:34] Bugs System
Pushed into mysql-5.1-telco-7.0 5.1.51-ndb-7.0.20 (revid:martin.skold@mysql.com-20101014082627-jrmy9xbfbtrebw3c) (version source revid:martin.skold@mysql.com-20101014082627-jrmy9xbfbtrebw3c) (merge vers: 5.1.51-ndb-7.0.20) (pib:21)
[14 Oct 2010 8:49] Bugs System
Pushed into mysql-5.1-telco-6.3 5.1.51-ndb-6.3.39 (revid:martin.skold@mysql.com-20101014083757-5qo48b86d69zjvzj) (version source revid:martin.skold@mysql.com-20101014083757-5qo48b86d69zjvzj) (merge vers: 5.1.51-ndb-6.3.39) (pib:21)
[14 Oct 2010 9:04] Bugs System
Pushed into mysql-5.1-telco-6.2 5.1.51-ndb-6.2.19 (revid:martin.skold@mysql.com-20101014084420-y54ecj85j5we27oa) (version source revid:martin.skold@mysql.com-20101014084420-y54ecj85j5we27oa) (merge vers: 5.1.51-ndb-6.2.19) (pib:21)
[14 Oct 2010 13:30] Jon Stephens
Already documented in the 5.1.50 changelog; no additional changelog entries required. Set back to Closed state.

Portions of this website are copyright © 2001, 2002 The PHP Group

Page generated in 0.051 sec. using MySQL 8.4.7-u6-cloud

Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the respective copyright holders.
It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.