Bug #69005 | Order by lower(column) produces out-of-order results | ||
---|---|---|---|
Submitted: | 19 Apr 2013 5:01 | Modified: | 11 Dec 2013 11:53 |
Reporter: | Bryan Turner | Email Updates: | |
Status: | Closed | Impact on me: | |
Category: | MySQL Server: Optimizer | Severity: | S3 (Non-critical) |
Version: | 5.6.11 | OS: | Linux (Ubuntu 12.04 LTS) |
Assigned to: | CPU Architecture: | Any | |
Tags: | regression |
[19 Apr 2013 5:01]
Bryan Turner
[19 Apr 2013 5:02]
Bryan Turner
Table dump as a standalone file
Attachment: tables.dmp (application/octet-stream, text), 3.94 KiB.
[19 Apr 2013 7:11]
Bryan Turner
In my testing, this same query returns results in the expected order under MySQL 5.6.10, as well as 5.5.29.
[19 Apr 2013 9:23]
MySQL Verification Team
Hello Bryan, Thank you for the report. Verified as described. Regards, Umesh
[19 Apr 2013 9:23]
MySQL Verification Team
### Affected mysql> select version(); +------------------+ | version() | +------------------+ | 5.6.11-debug-log | +------------------+ 1 row in set (0.00 sec) mysql> mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ from repository this_ order by lower(this_.name) asc limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.00 sec) mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ from repository this_ order by this_.name asc limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.00 sec)
[19 Apr 2013 9:24]
MySQL Verification Team
### Affected mysql> select version(); +------------------+ | version() | +------------------+ | 5.6.10-debug-log | +------------------+ 1 row in set (0.00 sec) mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ from repository this_ order by lower(this_.name) asc limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.01 sec) mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ from repository this_ order by this_.name asc limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.00 sec)
[19 Apr 2013 9:25]
MySQL Verification Team
#### Not affected mysql> select version(); +--------------+ | version() | +--------------+ | 5.5.31-debug | +--------------+ 1 row in set (0.00 sec) mysql> mysql> mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ from repository this_ order by lower(this_.name) asc limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.02 sec) mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ from repository this_ order by this_.name asc limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.00 sec)
[4 Jun 2013 17:45]
MySQL Verification Team
Testcase: ----------- drop table if exists `t1`; create table `t1`(`p` int,`n` char(3) charset utf8 collate utf8_bin)engine=myisam; insert into `t1` values (1,'a a'),(2,'a b'); select * from `t1` order by lower(`n`) asc; select * from `t1` order by lower(`n`) desc; ---------- Affects all versions of 5.6.x and 5.7.x
[16 Oct 2013 18:39]
Ben Stillman
Verified in 5.6.13 Enterprise on SmartOS (Solaris 11). uname -a SunOS 5.11 joyent_20130719T220136Z i86pc i386 i86pc Solaris mysql> select version (); +-------------------------------------------+ | version () | +-------------------------------------------+ | 5.6.13-enterprise-commercial-advanced-log | +-------------------------------------------+ 1 row in set (0.00 sec) mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, -> this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, -> this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ -> from repository this_ -> order by lower(this_.name) asc -> limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.00 sec) mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, -> this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, -> this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ -> from repository this_ -> order by this_.name asc -> limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.00 sec)
[16 Oct 2013 19:08]
Ben Stillman
It's the utf8_bin collation. A random sampling of other collations returns the correct order. mysql> ALTER TABLE repository CHANGE COLUMN name name VARCHAR(128) CHARACTER SET 'utf8' COLLATE 'utf8_general_ci' NOT NULL ; Query OK, 5 rows affected (0.02 sec) Records: 5 Duplicates: 0 Warnings: 0 mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, -> this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, -> this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ -> from repository this_ -> order by lower(this_.name) asc -> limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.00 sec)
[16 Oct 2013 19:26]
Ben Stillman
utf16_bin works as well: mysql> ALTER TABLE repository CHANGE COLUMN name name VARCHAR(128) CHARACTER SET 'utf16' COLLATE 'utf16_bin' NOT NULL ; Query OK, 5 rows affected (0.03 sec) Records: 5 Duplicates: 0 Warnings: 0 mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, -> this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, -> this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ -> from repository this_ -> order by lower(this_.name) asc -> limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.00 sec)
[7 Nov 2013 15:51]
Paul DuBois
Noted in 5.5.36, 5.6.15, 5.7.4 changelogs. For the utf8_bin collation, ORDER BY LOWER(col_name) could produce incorrect ordering.
[6 Dec 2013 19:19]
Bryan Turner
Now that 5.6.15 is available for download I have installed it clean VMs for both Ubuntu and Windows and this appears to _not_ be fixed. The same reproduce case I described when I created this issue still fails: mysql> select this_.id as id10_0_, this_.is_forkable as is2_10_0_, this_.hierarchy_id as hierarchy3_10_0_, this_.name as name10_0_, this_.origin_id as origin8_10_0_, this_.project_id as project9_10_0_, this_.scm_id as scm5_10_0_, this_.slug as slug10_0_, this_.state as state10_0_ from repository this_ order by lower(this_.name) asc limit 4; +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | id10_0_ | is2_10_0_ | hierarchy3_10_0_ | name10_0_ | origin8_10_0_ | project9_10_0_ | scm5_10_0_ | slug10_0_ | state10_0_ | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ | 3 | 0 | a | Cloned Repository | 1 | 2 | git | cloned_repository | 1 | | 5 | 0 | d | Restricted Repository | NULL | 4 | git | restricted_repo | 1 | | 4 | 1 | c | Repository with no pull requests | NULL | 2 | git | no_prs_repository | 1 | | 1 | 1 | a | Test Repository 1 | NULL | 1 | git | test_repository_1 | 1 | +---------+-----------+------------------+----------------------------------+---------------+----------------+------------+-------------------+------------+ 4 rows in set (0.00 sec) mysql> status; -------------- ./bin/mysql Ver 14.14 Distrib 5.6.15, for debian6.0 (x86_64) using EditLine wrapper Connection id: 1 Current database: oops Current user: root@localhost SSL: Not in use Current pager: stdout Using outfile: '' Using delimiter: ; Server version: 5.6.15 MySQL Community Server (GPL) ... Can someone please reconfirm this issue? Perhaps it is encoding specific. As I noted in my initial reproduce case, "oops" is: mysql> create database oops character set utf8 collate utf8_bin; Query OK, 1 row affected (0.00 sec) Thanks! Bryan Turner
[6 Dec 2013 19:26]
MySQL Verification Team
changelog entry could be wrong. it is fixed in bzr source, but not in 5.6.15. mysql> select * from `t1` order by lower(`n`) asc; +------+------+ | p | n | +------+------+ | 1 | a a | | 2 | a b | +------+------+ 2 rows in set (0.00 sec) mysql> select * from `t1` order by lower(`n`) desc; +------+------+ | p | n | +------+------+ | 2 | a b | | 1 | a a | +------+------+ 2 rows in set (0.00 sec) mysql> select version(); +--------------+ | version() | +--------------+ | 5.6.16-debug | +--------------+ 1 row in set (0.00 sec)
[6 Dec 2013 19:51]
Daniel Price
Correction: This bug is addressed in *5.6.16*, not 5.6.15 as previously stated. Noted in 5.5.36, *5.6.16*, 5.7.4 changelogs. For the utf8_bin collation, ORDER BY LOWER(col_name) could produce incorrect ordering.
[11 Dec 2013 11:53]
Erlend Dahl
Re-closing.
[2 Feb 2014 17:23]
Laurynas Biveinis
5.5$ bzr log -r 4541 ------------------------------------------------------------ revno: 4541 committer: Neeraj Bisht <neeraj.x.bisht@oracle.com> branch nick: 5.5 timestamp: Thu 2013-11-07 16:46:24 +0530 message: Bug#16691598 - ORDER BY LOWER(COLUMN) PRODUCES OUT-OF-ORDER RESULTS Problem:- We have created a table with UTF8_BIN collation. In case, when in our query we have ORDER BY clause over a function call we are getting result in incorrect order. Note:the bug is not there in 5.5. Analysis: In 5.5, for UTF16_BIN, we have min and max multi-byte length is 2 and 4 respectively.In make_sortkey(),for 2 byte character character we are assuming that the resultant length will be 2 byte/character. But when we use my_strnxfrm_unicode_full_bin(), we store sorting weights using 3 bytes per character.This result in truncated result. Same thing happen for UTF8MB4, where we have 1 byte min multi-byte and 4 byte max multi-byte.We will accsume resultant data as 1 byte/character, which result in truncated result. Solution:- use strnxfrm(means use of MY_CS_STRNXFRM macro) is used for sort, in which the resultant length is not dependent on source length.
[24 Apr 2014 23:35]
Jeremy Cole
This actually affects more than just LOWER(). It also at least affects ORDER BY c COLLATE utf8_bin amongst other things.