MySQL Bugs: #46080: group_concat(... order by) crashes server when sort_buffer

Bug #46080	group_concat(... order by) crashes server when sort_buffer_size cannot allocate
Submitted:	9 Jul 2009 12:19	Modified:	5 Aug 2009 0:01
Reporter:	Shane Bester (Platinum Quality Contributor)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: General	Severity:	S1 (Critical)
Version:	5.0.84, 5.1.37, 5.4.4	OS:	Any
Assigned to:	Georgi Kodinov	CPU Architecture:	Any
Tags:	group_concat, order by, sort_buffer_size

Description:
when using group_concat(... order by ..), mysql must allocate sort_buffer_size, but crashes in cases where this allocation cannot be made.  

5.1.37-debug stack trace:

mysqld.exe!Item_func_group_concat::add()[item_sum.cc:3335]
mysqld.exe!Item_sum::reset()[item_sum.h:290]
mysqld.exe!init_sum_functions()[sql_select.cc:15374]
mysqld.exe!end_send_group()[sql_select.cc:12151]
mysqld.exe!do_select()[sql_select.cc:10871]
mysqld.exe!JOIN::exec()[sql_select.cc:2199]
mysqld.exe!mysql_select()[sql_select.cc:2388]
mysqld.exe!handle_select()[sql_select.cc:268]
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:5013]
mysqld.exe!mysql_execute_command()[sql_parse.cc:2208]
mysqld.exe!mysql_parse()[sql_parse.cc:5933]
mysqld.exe!dispatch_command()[sql_parse.cc:1213]
mysqld.exe!do_command()[sql_parse.cc:854]
mysqld.exe!handle_one_connection()[sql_connect.cc:1127]
mysqld.exe!pthread_start()[my_winthread.c:85]
mysqld.exe!_callthreadstart()[thread.c:293]
mysqld.exe!_threadstart()[thread.c:277]
kernel32.dll!FlsSetValue()

Please note, this bug affects both debug and release builds, but the testcase requires full debug binary to simulate the low memory condition.

How to repeat:
build full debug server, and start it like this to simulate low memory:

mysqld --skip-grant-tables --skip-name-resolve --console 
--safemalloc-mem-limit=8000000 --skip-innodb

Now run this SQL:

drop table if exists t1;
create table t1(a char(255))engine=myisam;
insert into t1 values ('a');
set session sort_buffer_size=64*1024*1024;
select group_concat(a order by a) from t1 group by a;

Suggested fix:
fix the mysql code in Item_func_group_concat::add() to check the return value of tree_insert function is not null.

Thank you for the bug report. Verified as described however I wasn't able to repeat with 6.0.12.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78383

2800 Georgi Kodinov	2009-07-10
      Bug #46080: group_concat(... order by) crashes server when
        sort_buffer_size cannot allocate
      
      The NULL return from tree_insert() (on low memory) was not
      checked for in Item_func_group_concat::add(). As a result
      on low memory conditions a crash happens.
      
      Fixed by properly checking the return code.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78409

2802 Georgi Kodinov	2009-07-10
      Bug #46080: group_concat(... order by) crashes server when
        sort_buffer_size cannot allocate
      
      The NULL return from tree_insert() (on low memory) was not
      checked for in Item_func_group_concat::add(). As a result
      on low memory conditions a crash happens.
      
      Fixed by properly checking the return code.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78413

2803 Georgi Kodinov	2009-07-10
      Addendum to the fix for bug #46080: fixed the test case

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/78523

2804 Georgi Kodinov	2009-07-13
      Addendum to the fix for bug #46080: fixed the error handling

Pushed into 5.0.85 (revid:joro@sun.com-20090713173958-lgg416vhsv6rpgb8) (version source revid:joro@sun.com-20090713111714-8ff86iblx722iubf) (merge vers: 5.0.85) (pib:11)

Pushed into 5.1.37 (revid:joro@sun.com-20090713174543-cd2x7q1gi1hzoand) (version source revid:joro@sun.com-20090713113423-nu88fbv2q80pgx3x) (merge vers: 5.1.37) (pib:11)

Noted in 5.0.85, 5.1.37 changelogs.

The server crashed if evaluation of GROUP_CONCAT(... ORDER BY)
required allocation of a sort buffer but allocation failed. 

Setting report to NDI pending push into 5.4.x.

Pushed into 5.4.4-alpha (revid:alik@sun.com-20090804194615-h40sa098mx4z49qg) (version source revid:joro@sun.com-20090713113640-7d6k13z12pj5eu13) (merge vers: 5.4.4-alpha) (pib:11)

Noted in 5.4.4 changelog.

Noted in 5.4.2 changelog because next 5.4 version will be 5.4.2 and not 5.4.4.

Ignore previous comment about 5.4.2.

Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)

Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)

Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

The 5.4 fix has been pushed to 5.4.2.