Bug #29610 func_group fails with lost connection

Description:
Test case func_group fails in normal test runs on AIX 5.2, HPUX 11.00, 11.11, 11.23, Solaris Sparc, Linux S390, Mac OS X Tiger PPC:

func_group                     [ fail ]

Errors are (from /PATH/mysqltest-time) :
mysqltest: At line NNN: query 'SELECT a,COUNT(DISTINCT b) AS cnt FROM t1 GROUP BY a HAVING cnt > 50' failed: 2013: Lost connection to MySQL server during query
(the last lines may be the most important ones)
Result from queries before failure can be found in /PATH/mysql-test/var/log/func_group.log

And in embedded tests:
func_group                     [ fail ]

ERROR: mysqltest returned unexpected code 139, it has probably crashed

Or:
func_group                     [ fail ]

ERROR: mysqltest returned unexpected code 138, it has probably crashed

How to repeat:
Build 5.1.20 and run test on mentioned platforms.

Additional note: Appears to occur on 64-bit platforms only.

Thank you for the report.

Verified as described.

Ok. managed to repeat. Stack trace:

#0  0x000000010044078c in ptr_compare_1 (compare_length=0xffffffff7d33d494, 
    a=0x1009c9a80, b=0x1009c9a80) at ptr_cmp.c:92
#1  0x0000000100441dd8 in queue_insert (queue=0xffffffff7d33d420, 
    element=0x1009c9a70 "") at queues.c:216
#2  0x00000001002e4d34 in merge_buffers (param=0xffffffff7d33d6e0, 
    from_file=0xffffffff7d33d870, to_file=0x100965ed0, 
    sort_buffer=0x100979a50 "\001\200", lastbuff=0x1009c9a70, Fb=0x1009c9a70, 
    Tb=0x1009c9b60, flag=1) at filesort.cc:1178
#3  0x00000001002e564c in merge_index (param=0xffffffff7d33d6e0, 
    sort_buffer=0x100979a50 "\001\200", buffpek=0x1009c9a70, maxbuffer=5, 
    tempfile=0xffffffff7d33d870, outfile=0x100965ed0) at filesort.cc:1329
#4  0x00000001002e2354 in filesort (thd=0x100949a70, table=0x100966a40, 
    sortorder=0x100963d80, s_length=1, select=0x100963818, 
    max_rows=18446744073709551615, sort_positions=false, 
    examined_rows=0xffffffff7d33dac0) at filesort.cc:275
#5  0x000000010024f580 in create_sort_index (thd=0x100949a70, 
    join=0x100967a60, order=0x100962a48, filesort_limit=18446744073709551615, 
    select_limit=18446744073709551615, is_order_by=true) at sql_select.cc:12905
#6  0x000000010023004c in JOIN::exec (this=0x100967a60) at sql_select.cc:2040
#7  0x0000000100230a7c in mysql_select (thd=0x100949a70, 
    rref_pointer_array=0x10094b5b0, tables=0x1009625e8, wild_num=0, 
    fields=@0x10094b4d0, conds=0x0, og_num=1, order=0x0, group=0x100962a48, 
    having=0x100962be8, proc_param=0x0, select_options=2147764736, 
---Type <return> to continue, or q <return> to quit---
    result=0x100962e20, unit=0x10094afc8, select_lex=0x10094b3c8)
    at sql_select.cc:2273
#8  0x0000000100228b30 in handle_select (thd=0x100949a70, lex=0x10094af28, 
    result=0x100962e20, setup_tables_done_option=0) at sql_select.cc:246
#9  0x00000001001d0858 in execute_sqlcom_select (thd=0x100949a70, 
    all_tables=0x1009625e8) at sql_parse.cc:4493
#10 0x00000001001c7fc8 in mysql_execute_command (thd=0x100949a70)
    at sql_parse.cc:1838
#11 0x00000001001d2b44 in mysql_parse (thd=0x100949a70, 
    inBuf=0x100962060 "SELECT a,COUNT(DISTINCT b) AS cnt FROM t1 GROUP BY a HAVING cnt > 50", length=68, found_semicolon=0xffffffff7d33fc58)
    at sql_parse.cc:5395

#0  0x000000010044078c in ptr_compare_1 (compare_length=0xffffffff7d33d494, 
    a=0x1009c9a80, b=0x1009c9a80) at ptr_cmp.c:92
92        reg3 int length= *compare_length-1;
(gdb) p *compare_length
$2 = 21474836494
(gdb) p/x *compare_length
$3 = 0x50000000e

^^  this seems to be the problem

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/30668

ChangeSet@1.2540, 2007-07-11 15:16:54+04:00, sergefp@mysql.com +1 -0
  BUG#29610: crash in func_group on 64bit platform:
  - make merge_buffers():sort_length have type size_t as this type is 
    expected by, e.g. ptr_compare_1, which will receive pointer to 
    sort_length as comparison parameter.

Pushed into 5.1.21-beta

Yes:
On 64-bit platforms, filesort code (i.e. GROUP BY/ORDER BY queries) could cause crash.

Noted in 5.1.21 changelog.

On 64-bit platforms, the filesort code (for queries with GROUP BY or
ORDER BY) could crash due to an incorrect pointer size.