Bug #46080 group_concat(... order by) crashes server when sort_buffer_size cannot allocate
Submitted: 9 Jul 2009 12:19 Modified: 5 Aug 2009 0:01
Reporter: Shane Bester (Platinum Quality Contributor) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: General Severity:S1 (Critical)
Version:5.0.84, 5.1.37, 5.4.4 OS:Any
Assigned to: Georgi Kodinov
Tags: group_concat, order by, sort_buffer_size
Triage: Triaged: D1 (Critical)

[9 Jul 2009 12:19] Shane Bester
Description:
when using group_concat(... order by ..), mysql must allocate sort_buffer_size, but crashes in cases where this allocation cannot be made.  

5.1.37-debug stack trace:

mysqld.exe!Item_func_group_concat::add()[item_sum.cc:3335]
mysqld.exe!Item_sum::reset()[item_sum.h:290]
mysqld.exe!init_sum_functions()[sql_select.cc:15374]
mysqld.exe!end_send_group()[sql_select.cc:12151]
mysqld.exe!do_select()[sql_select.cc:10871]
mysqld.exe!JOIN::exec()[sql_select.cc:2199]
mysqld.exe!mysql_select()[sql_select.cc:2388]
mysqld.exe!handle_select()[sql_select.cc:268]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:5013]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2208]
mysqld.exe!mysql_parse()[sql_parse.cc:5933]
mysqld.exe!dispatch_command()[sql_parse.cc:1213]
mysqld.exe!do_command()[sql_parse.cc:854]
mysqld.exe!handle_one_connection()[sql_connect.cc:1127]
mysqld.exe!pthread_start()[my_winthread.c:85]
mysqld.exe!_callthreadstart()[thread.c:293]
mysqld.exe!_threadstart()[thread.c:277]
kernel32.dll!FlsSetValue()

Please note, this bug affects both debug and release builds, but the testcase requires full debug binary to simulate the low memory condition.

How to repeat:
build full debug server, and start it like this to simulate low memory:

mysqld --skip-grant-tables --skip-name-resolve --console 
--safemalloc-mem-limit=8000000 --skip-innodb

Now run this SQL:

drop table if exists t1;
create table t1(a char(255))engine=myisam;
insert into t1 values ('a');
set session sort_buffer_size=64*1024*1024;
select group_concat(a order by a) from t1 group by a;

Suggested fix:
fix the mysql code in Item_func_group_concat::add() to check the return value of tree_insert function is not null.
[9 Jul 2009 14:51] Miguel Solorzano
Thank you for the bug report. Verified as described however I wasn't able to repeat with 6.0.12.
[10 Jul 2009 12:00] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78383

2800 Georgi Kodinov	2009-07-10
      Bug #46080: group_concat(... order by) crashes server when
        sort_buffer_size cannot allocate
      
      The NULL return from tree_insert() (on low memory) was not
      checked for in Item_func_group_concat::add(). As a result
      on low memory conditions a crash happens.
      
      Fixed by properly checking the return code.
[10 Jul 2009 13:45] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78409

2802 Georgi Kodinov	2009-07-10
      Bug #46080: group_concat(... order by) crashes server when
        sort_buffer_size cannot allocate
      
      The NULL return from tree_insert() (on low memory) was not
      checked for in Item_func_group_concat::add(). As a result
      on low memory conditions a crash happens.
      
      Fixed by properly checking the return code.
[10 Jul 2009 14:03] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78413

2803 Georgi Kodinov	2009-07-10
      Addendum to the fix for bug #46080: fixed the test case
[13 Jul 2009 11:18] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78523

2804 Georgi Kodinov	2009-07-13
      Addendum to the fix for bug #46080: fixed the error handling
[13 Jul 2009 17:42] Bugs System
Pushed into 5.0.85 (revid:joro@sun.com-20090713173958-lgg416vhsv6rpgb8) (version source revid:joro@sun.com-20090713111714-8ff86iblx722iubf) (merge vers: 5.0.85) (pib:11)
[13 Jul 2009 17:48] Bugs System
Pushed into 5.1.37 (revid:joro@sun.com-20090713174543-cd2x7q1gi1hzoand) (version source revid:joro@sun.com-20090713113423-nu88fbv2q80pgx3x) (merge vers: 5.1.37) (pib:11)
[13 Jul 2009 19:35] Paul Dubois
Noted in 5.0.85, 5.1.37 changelogs.

The server crashed if evaluation of GROUP_CONCAT(... ORDER BY)
required allocation of a sort buffer but allocation failed. 

Setting report to NDI pending push into 5.4.x.
[4 Aug 2009 19:51] Bugs System
Pushed into 5.4.4-alpha (revid:alik@sun.com-20090804194615-h40sa098mx4z49qg) (version source revid:joro@sun.com-20090713113640-7d6k13z12pj5eu13) (merge vers: 5.4.4-alpha) (pib:11)
[5 Aug 2009 0:01] Paul Dubois
Noted in 5.4.4 changelog.
[12 Aug 2009 22:09] Paul Dubois
Noted in 5.4.2 changelog because next 5.4 version will be 5.4.2 and not 5.4.4.
[14 Aug 2009 23:00] Paul Dubois
Ignore previous comment about 5.4.2.
[26 Aug 2009 13:45] Bugs System
Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)
[26 Aug 2009 13:46] Bugs System
Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)
[26 Aug 2009 13:48] Bugs System
Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)
[27 Aug 2009 16:32] Bugs System
Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)
[7 Oct 2009 16:23] Paul Dubois
The 5.4 fix has been pushed to 5.4.2.