Bug #46448 trailing spaces are not ignored when user collation maps space != 0x20
Submitted: 29 Jul 2009 12:03 Modified: 19 Dec 2009 0:05
Reporter: Axel Schwenke Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Charsets Severity:S2 (Serious)
Version:5.0.74 OS:Any
Assigned to: V Venkateswaran
Tags: regression
Triage: Triaged: D3 (Medium)

[29 Jul 2009 12:03] Axel Schwenke
Description:
When a user defined collation maps the space character different than 0x20 then trailing spaces will no longer be ignored in string comparisons.

This used to work in 5.0.36 but no longer in 5.0.74.

How to repeat:
Add another collation, i.e. latin1_test from the attached files. Then:

mysql>select "foo" = "foo " collate latin1_test;
+------------------------------------+
| "foo" = "foo " collate latin1_test |
+------------------------------------+
|                                  0 | 
+------------------------------------+
1 row in set (0,00 sec)

mysql>select "foo" = "foo " collate latin1_bin;
+-----------------------------------+
| "foo" = "foo " collate latin1_bin |
+-----------------------------------+
|                                 1 | 
+-----------------------------------+
1 row in set (0,01 sec)

Suggested fix:
The problem seems to be in line 182 in ctype-simple.c:

180 for (end= a + a_length-length; a < end ; a++)
181 {
182     if (map[*a] != ' ')
183         return (map[*a] < ' ') ? -swap : swap;
184 }

here the mapped character is compared with a literal space character; should be

180 for (end= a + a_length-length; a < end ; a++)
181 {
182     if (map[*a] != map[' '])
183         return (map[*a] < map[' ']) ? -swap : swap;
184 }
[17 Sep 2009 12:00] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/83592

3105 V Narayanan	2009-09-17
      Bug#46448 trailing spaces are not ignored when user collation maps space != 0x20
      
      In MySQL when the mapping for space is changed to something other than
      0x20 by defining a different collation, then space is not ignored when
      comparing two strings.
      
      This was happening because the function that performs the comparison
      of two strings while ignoring ending spaces, was comparing the collation
      value of a space with the ascii value of the ' ' character. This should
      be changed to do comparison between the collated values.
     @ strings/ctype-simple.c
        Bug#46448 trailing spaces are not ignored when user collation maps space != 0x20
        
        change my_strnncollsp_simple to compare collated values when checking
        for trailing spaces. Currently the comparison happens between a collated
        value and the ascii value.
[18 Sep 2009 10:09] V Venkateswaran
The patch in http://lists.mysql.com/commits/83592 has been made in accordance with solution suggested by Axel
[24 Sep 2009 8:09] Alexander Barkov
Hi Narayanan,

I have two concerns about the patch:

1. I think the correct code should look like this:

180 for (end= a + a_length-length; a < end ; a++)
181 {
182     if (*a != ' ')
183         return (map[*a] < map[' ']) ? -swap : swap;
184 }

We ignore only *spaces*, but I am not sure that we should 
ignore characters that are sorted equal to space.

2. Please add a test covering this patch. Have a look into
ctype_ldml.test and std_data/Index.xml to check how to test
a user defined collation.
[24 Sep 2009 8:30] Alexander Barkov
All right, it seems that the assumption that all trailing characters equal to
space should be ignored was previously made when fixing Bug#29461 and Bug#29261.
So the patch looks correct.

Please add a test case only, and commit a new patch.

Thanks.
[12 Oct 2009 7:43] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86527
[12 Oct 2009 7:53] Alexander Barkov
http://lists.mysql.com/commits/86527 is ok to push.
[12 Oct 2009 9:36] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86534

3160 V Narayanan	2009-10-12
      Bug#46448 trailing spaces are not ignored when user collation maps space != 0x20
      
      Fixing copyright header in test collation file.
     @ mysql-test/std_data/latin1.xml
        Bug#46448 trailing spaces are not ignored when user collation maps space != 0x20
        
        Fixing copy right header in test collation file.
[12 Oct 2009 9:56] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86538

3161 V Narayanan	2009-10-12
      Bug#46448 trailing spaces are not ignored when user collation maps space != 0x20
      
      changing year in copyright header to 2009.
     @ mysql-test/std_data/latin1.xml
        Bug#46448 trailing spaces are not ignored when user collation maps space != 0x20
        
        changing year in copyright header to 2009.
[13 Oct 2009 17:40] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/86715

3653 V Narayanan	2009-10-13
      Bug#46448 trailing spaces are not ignored when user collation maps space != 0x20
      
      Fix test case to list newly added test collation.
     @ mysql-test/r/innodb_ctype_ldml.result
        Bug#46448 trailing spaces are not ignored when user collation maps space != 0x20
        
        Fix test case to list newly added test collation.
[14 Oct 2009 14:39] Bugs System
Pushed into 5.1.41 (revid:joro@sun.com-20091014143611-cphb0enjlx6lpat1) (version source revid:satya.bn@sun.com-20091013071829-zc4c3go44j6re592) (merge vers: 5.1.40) (pib:13)
[22 Oct 2009 6:36] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091022063126-l0qzirh9xyhp0bpc) (version source revid:alik@sun.com-20091019135554-s1pvptt6i750lfhv) (merge vers: 6.0.14-alpha) (pib:13)
[22 Oct 2009 7:08] Bugs System
Pushed into 5.5.0-beta (revid:alik@sun.com-20091022060553-znkmxm0g0gm6ckvw) (version source revid:alik@sun.com-20091019131708-bc6pv55x6287a0wc) (merge vers: 5.5.0-beta) (pib:13)
[22 Oct 2009 11:48] Axel Schwenke
Hint for docs team: this is a regression from bug #29461, introduced in 5.0.48
[28 Oct 2009 22:12] Paul Dubois
Noted in 5.1.41, 5.5.0, 6.0.14 changelogs.

Trailing spaces were not ignored for user-defined collations that
mapped spaces to a character other than 0x20.
[18 Dec 2009 10:37] Bugs System
Pushed into 5.1.41-ndb-7.1.0 (revid:jonas@mysql.com-20091218102229-64tk47xonu3dv6r6) (version source revid:jonas@mysql.com-20091218095730-26gwjidfsdw45dto) (merge vers: 5.1.41-ndb-7.1.0) (pib:15)
[18 Dec 2009 10:53] Bugs System
Pushed into 5.1.41-ndb-6.2.19 (revid:jonas@mysql.com-20091218100224-vtzr0fahhsuhjsmt) (version source revid:jonas@mysql.com-20091217101452-qwzyaig50w74xmye) (merge vers: 5.1.41-ndb-6.2.19) (pib:15)
[18 Dec 2009 11:07] Bugs System
Pushed into 5.1.41-ndb-6.3.31 (revid:jonas@mysql.com-20091218100616-75d9tek96o6ob6k0) (version source revid:jonas@mysql.com-20091217154335-290no45qdins5bwo) (merge vers: 5.1.41-ndb-6.3.31) (pib:15)
[18 Dec 2009 11:22] Bugs System
Pushed into 5.1.41-ndb-7.0.11 (revid:jonas@mysql.com-20091218101303-ga32mrnr15jsa606) (version source revid:jonas@mysql.com-20091218064304-ezreonykd9f4kelk) (merge vers: 5.1.41-ndb-7.0.11) (pib:15)