Bug #49897 crash in ptr_compare when char(0) NOT NULL column is used for ORDER BY
Submitted: 23 Dec 2009 19:16 Modified: 12 Mar 2010 16:25
Reporter: Matthew Lord Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: General Severity:S1 (Critical)
Version:5.1.41, 5.0 OS:Any
Assigned to: Ramil Kalimullin CPU Architecture:Any
Tags: crash, order by, ptr_compare, sort_buffer_size
Triage: Triaged: D1 (Critical)

[23 Dec 2009 19:16] Matthew Lord
Description:
#4  <signal handler called>
#5  0x084cc9ff in ptr_compare (compare_length=0xecdb794, a=0xba153ec, b=0xba153ec) at ptr_cmp.c:62

How to repeat:
I have the data that can be used to repeat this.  Let me know if you need it.

Suggested fix:
We need to accommodate for this situation here:

 while (--length)
  {
    if (*first++ != *last++)  // <---- crash occurs here with a char(0)
                              // NOT NULL column
      return (int) first[-1] - (int) last[-1];
  }
  return (int) first[0] - (int) last[0];
[24 Dec 2009 16:28] Shane Bester
stack trace from 5.1.41:
mysqld.exe!ptr_compare()[ptr_cmp.c:62]
mysqld.exe!queue_insert()[queues.c:218]
mysqld.exe!merge_buffers()[filesort.cc:1243]
mysqld.exe!merge_index()[filesort.cc:1395]
mysqld.exe!filesort()[filesort.cc:290]
mysqld.exe!create_sort_index()[sql_select.cc:13646]
mysqld.exe!JOIN::exec()[sql_select.cc:2195]
mysqld.exe!mysql_select()[sql_select.cc:2437]
mysqld.exe!handle_select()[sql_select.cc:269]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:5052]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2246]
mysqld.exe!mysql_parse()[sql_parse.cc:5974]
mysqld.exe!dispatch_command()[sql_parse.cc:1233]
mysqld.exe!do_command()[sql_parse.cc:872]
mysqld.exe!handle_one_connection()[sql_connect.cc:1127]
mysqld.exe!pthread_start()[my_winthread.c:85]
mysqld.exe!_callthreadstart()[thread.c:295]
mysqld.exe!_threadstart()[thread.c:275]
kernel32.dll!BaseThreadStart()
[24 Dec 2009 20:01] Shane Bester
perhaps related to bug #48617
[16 Jan 2010 6:06] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/97169

2841 Ramil Kalimullin	2010-01-16
      Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
      column is used for ORDER BY
      
      Problem: filesort isn't meant for null length sort data
      (e.g. char(0)), that leads to a server crash.
      
      Fix: disregard sort order if sort data record length is 0 (nothing
      to sort).
     @ mysql-test/r/select.result
        Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
        column is used for ORDER BY
          - test result.
     @ mysql-test/t/select.test
        Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
        column is used for ORDER BY
          - test case.
     @ sql/filesort.cc
        Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
        column is used for ORDER BY
          - assert added as filesort cannot handle null length sort data.
     @ sql/sql_select.cc
        Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
        column is used for ORDER BY
          - don't sort null length data e.g. in case of ORDER BY CHAR(0).
[29 Jan 2010 9:18] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/98550

2843 Ramil Kalimullin	2010-01-29
      Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
      column is used for ORDER BY
      
      Problem: filesort isn't meant for null length sort data
      (e.g. char(0)), that leads to a server crash.
      
      Fix: disregard sort order if sort data record length is 0 (nothing
      to sort).
     @ mysql-test/r/select.result
        Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
        column is used for ORDER BY
          - test result.
     @ mysql-test/t/select.test
        Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
        column is used for ORDER BY
          - test case.
     @ sql/filesort.cc
        Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
        column is used for ORDER BY
          - assert added as filesort cannot handle null length sort data.
     @ sql/sql_select.cc
        Fix for bug#49897: crash in ptr_compare when char(0) NOT NULL 
        column is used for ORDER BY
          - don't sort null length data e.g. in case of ORDER BY CHAR(0).
[1 Feb 2010 11:41] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/98809

2845 Georgi Kodinov	2010-02-01
      fixed a typo in bug #49897.
[4 Feb 2010 10:15] Bugs System
Pushed into 5.0.91 (revid:joro@sun.com-20100204101329-4wg1ktw00vk63o8l) (version source revid:joro@sun.com-20100201114016-jylx4hivgqbs0vg2) (merge vers: 5.0.90) (pib:16)
[4 Feb 2010 10:19] Bugs System
Pushed into 5.1.44 (revid:joro@sun.com-20100204101444-2j32mhqroo0iiio6) (version source revid:joro@sun.com-20100201115030-hgvq6489bt0w3rty) (merge vers: 5.1.43) (pib:16)
[5 Feb 2010 11:47] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100204063540-9czpdmpixi3iw2yb) (version source revid:alik@sun.com-20100203162117-gjiiuzj6sq2ohlss) (pib:16)
[5 Feb 2010 11:56] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100205113942-oqovjy0eoqbarn7i) (version source revid:alik@sun.com-20100204064210-ljwanqvrjs83s1gq) (merge vers: 6.0.14-alpha) (pib:16)
[5 Feb 2010 11:59] Bugs System
Pushed into 5.5.2-m2 (revid:alik@sun.com-20100203172258-1n5dsotny40yufxw) (version source revid:alik@sun.com-20100203140148-nmlve92a9cq69vp9) (merge vers: 5.5.2-m2) (pib:16)
[11 Feb 2010 22:01] Paul Dubois
Noted in 5.0.91, 5.1.44, 5.5.2, 6.0.14 changelogs.

The filesort sorting method applied to a CHAR(0) column could lead to
a server crash. 

Setting report to Need Merge pending push to Celosia.
[12 Mar 2010 14:16] Bugs System
Pushed into 5.1.44-ndb-7.0.14 (revid:jonas@mysql.com-20100312135944-t0z8s1da2orvl66x) (version source revid:jonas@mysql.com-20100312115609-woou0te4a6s4ae9y) (merge vers: 5.1.44-ndb-7.0.14) (pib:16)
[12 Mar 2010 14:32] Bugs System
Pushed into 5.1.44-ndb-6.2.19 (revid:jonas@mysql.com-20100312134846-tuqhd9w3tv4xgl3d) (version source revid:jonas@mysql.com-20100312060623-mx6407w2vx76h3by) (merge vers: 5.1.44-ndb-6.2.19) (pib:16)
[12 Mar 2010 14:48] Bugs System
Pushed into 5.1.44-ndb-6.3.33 (revid:jonas@mysql.com-20100312135724-xcw8vw2lu3mijrhn) (version source revid:jonas@mysql.com-20100312103652-snkltsd197l7q2yg) (merge vers: 5.1.44-ndb-6.3.33) (pib:16)
[12 Mar 2010 16:25] Paul Dubois
Already fixed in earlier 5.1.x, 5.5.x.
[9 Apr 2010 14:51] Paul Dubois
Noted in 5.1.43sp1 changelog.