Bug #11385 crash in group by and DIV

Description:
checked in mysql 5.0.7 linux/x86 and 4.1.11 windows/x86

the mysqld is crashing on the following query:

select dt DIV 10 as f, id from ts group by f;

valgrind says:
==17648== Thread 11:
==17648== Invalid write of size 4
==17648==    at 0x83B5427: memcpy (../sysdeps/generic/memcpy.c:55)
==17648==    by 0x8376A5B: my_longlong10_to_str_8bit (ctype-simple.c:913)
==17648==    by 0x808E87E: Field_longlong::val_str(String *, String *) (in /usr/local/mysql-debug-5.0.7-beta-linux-i686/bin/mysqld)
==17648==    by 0x809F1D1: Protocol_simple::store(Field *) (in /usr/local/mysql-debug-5.0.7-beta-linux-i686/bin/mysqld)
==17648==    by 0x8053F1B: Item_field::send(Protocol *, String *) (item.cc:3915)
==17648==    by 0x8099204: select_send::send_data(List<Item> &) (sql_class.cc:877)
==17648==    by 0x80FA888: end_send(JOIN *, st_join_table *, bool) (sql_select.cc:10045)
==17648==    by 0x80F92F8: evaluate_join_record(JOIN *, st_join_table *, int, char *) (sql_select.cc:9376)
==17648==    by 0x80F912B: sub_select(JOIN *, st_join_table *, bool) (sql_select.cc:9267)
==17648==    by 0x80F8C62: do_select(JOIN *, List<Item> *, st_table *, Procedure *) (sql_select.cc:9024)
==17648==    by 0x80E90E3: JOIN::exec(void) (sql_select.cc:1649)
==17648==    by 0x80E9D70: mysql_select(THD *, Item ***, st_table_list *, unsigned int, List<Item> &, Item *, unsigned int, st_order *, st_order *, Item *, st_order *, unsigned long, select_result *, st_select_lex_unit *, st_select_lex *) (sql_select.cc:2043)
==17648==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

How to repeat:
create table ts ( id int, dt datetime ) engine = innodb;

insert into id values ( 1, now() );
insert into id values ( 1, now() );
insert into id values ( 1, now() );
insert into id values ( 1, now() );

select dt DIV 10 as f, id from ts group by f;

Suggested fix:
don't crash

This bug was investigated by Alexander Ivanov. Here's his report.

The cause of the crash is as follows. The Field_datetime class inherits the decimals() function from the Field_Str class. This function simply returns NOT_FIXED_DEC = 0x1F value. Within the reported example when the Item_func_int_div::fix_length_and_dec() function is called:

void Item_func_int_div::fix_length_and_dec()
{
  find_num_type();
  max_length=args[0]->max_length - args[0]->decimals;
  maybe_null=1;
}

we obtain:

args[0]->max_length = 0x13
args[0]->decimals   = 0x1F
max_length          = 0xFFFFFFF4

Later this leads to allocating ALIGN_SIZE(max_length + 1) = 0xFFFFFFF8 bytes.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/26210

This patch was prepared by Alexander Ivanov.

The bug was due to the fact that method decimals was not defined for
class Field_datatime, and as a result was inherited from Field_str.

ChangeSet
  1.2302 05/06/20 10:49:04 igor@rurik.mysql.com +3 -0
  group_by.result:
    Added  a test case for bug #11385.
  group_by.test:
    Added  a test case for bug #11385.
  field.h:
    Fixed bug #11385.
    The bug was due to not defined method decimals for the class
    Field_datetime.

The fix will appear in 4.1.13 and 5.0.8.

Documented in the 4.1.13 and 5.0.8 changelogs.