Bug #25804 Crash with group_concat() and InnoDB
Submitted: 23 Jan 2007 21:52 Modified: 30 Jan 2007 1:34
Reporter: Pete Harlan (Candidate Quality Contributor) Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version:5.0.33, 5.0.27 OS:Linux (Linux x86_64 (Debian Etch))
Assigned to: Assigned Account CPU Architecture:Any

[23 Jan 2007 21:52] Pete Harlan
Description:
The following script crashes the 5.0.33 server.  Compiled from source under x86_64, with the following:

CC=gcc					\
    CFLAGS="-O2				\
    CXX=gcc				\
    CXXFLAGS="-O2 -felide-constructors -fno-exceptions -fno-rtti"	\
    ./configure	--prefix=/usr/local/mysql	\
        --exec-prefix=/usr/local/mysql		\
        --enable-assembler			\
        --with-debug=full			\
        --enable-thread-safe-client		\
        --with-extra-charsets=complex		\
        --disable-shared			\
        --with-mysqld-ldflags=-all-static

I also tried without debugging and with -O3.

gcc version: gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
Also fails when compiled with gcc version 3.4.6.
glibc version: 2.3.6
kernel version: 2.6.17.11 and 2.6.19.2

How to repeat:
Execute the following SQL.  Changing table types to MyISAM works okay; it fails only with InnoDB, at least in this test.

DROP TABLE IF EXISTS foo;
CREATE TABLE foo (
  id int(11) NOT NULL,
  gi int(11) NOT NULL,
  gt varchar(5) NOT NULL,
  p int(11) NOT NULL,
  PRIMARY KEY  (gt,gi,id)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

INSERT INTO foo VALUES (170,13674,'c',2);

DROP TABLE IF EXISTS bar;
CREATE TABLE bar (
  gt varchar(5) NOT NULL,
  gi int(11) NOT NULL,
  s varchar(10) NOT NULL,
  nr int(11) NOT NULL,
  n int(11) NOT NULL,
  dt int(11) NOT NULL,
  PRIMARY KEY  (gt,gi)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

INSERT INTO bar VALUES ('c',13674,'13834a0730',0,0,2007);

select  bar.s,
        group_concat(bar.gt),
        min(bar.dt) as minDT,
        min(foo.p) as minP,
        sum(bar.nr) as sumNR,
        sum(bar.n) as sumN,
        min(bar.gi) as minGI
from    foo join bar using (gt, gi)
where   foo.id  = 170
group by bar.s
order by minDT, minP desc, sumNR, sumN, bar.gt desc

Suggested fix:
Thank you.
[23 Jan 2007 21:56] Pete Harlan
I forgot to include the error-log's contents.  Here it is.  Please let me know if I can be of any assistance.  --Pete

070123 13:49:19 [Note] /usr/local/mysql/libexec/mysqld: ready for connections.
Version: '5.0.33-debug-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  
Source distribution
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=5242880
read_buffer_size=1044480
max_used_connections=1
max_connections=400
threads_connected=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 822705
 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x2bcf428
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x44088e78, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x6e616c726168
New value of fp=0x2bcf428 failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow 
instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x2c0e908 = select        bar.s,
                group_concat(bar.gt),
                min(bar.dt) as minDT,
                min(foo.p) as minP,
                sum(bar.nr) as sumNR,
                sum(bar.n) as sumN,
                min(bar.gi) as minGI
from    foo join bar using (gt, gi)
where   foo.id  = 170
group by bar.s
order by minDT, minP desc, sumNR, sumN, bar.gt desc
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
[23 Jan 2007 22:01] Pete Harlan
Also crashes the MySQL-compiled 5.0.27 x86-64 binary.  Does not crash 4.1.20.
[24 Jan 2007 1:54] MySQL Verification Team
Back trace for current source server on Ubuntu 6.10 32-bit:

070123 23:52:56 [Note] /home/miguel/dbs/5.0/libexec/mysqld: ready for connections.
Version: '5.0.36-debug'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
[New Thread -1291838560 (LWP 21128)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1291838560 (LWP 21128)]
0x0822dc2e in make_unireg_sortorder (order=0x8c8e3e0, length=0xb30002f4, sortorder=0x8c962f0) at sql_select.cc:12702
12702       if (order->item[0]->type() == Item::FIELD_ITEM)
(gdb) bt full
#0  0x0822dc2e in make_unireg_sortorder (order=0x8c8e3e0, length=0xb30002f4, sortorder=0x8c962f0) at sql_select.cc:12702
        count = 5
        sort = <value optimized out>
        pos = <value optimized out>
        _db_func_ = 0xb30002a4 "è\002"
        _db_file_ = 0xb30002a8 "ä\002"
        _db_level_ = 6219
        _db_framep_ = (char **) 0x8242092
#1  0x0823e13b in create_sort_index (thd=0x8c535e8, join=0x8c92678, order=0x8c8dff8, filesort_limit=18446744073709551615, 
    select_limit=18446744073709551615) at sql_select.cc:12287
        length = <value optimized out>
        examined_rows = <value optimized out>
        table = (TABLE *) 0x8cb61d0
        select = (SQL_SELECT *) 0x0
        tab = (JOIN_TAB *) 0x8c96180
        _db_func_ = 0x8c538c0 "(áR\b"
        _db_file_ = 0x8230da4 "\200x("
        _db_level_ = 3003122440
        _db_framep_ = (char **) 0xb30002f0
#2  0x0824cc76 in JOIN::exec (this=0x8c92678) at sql_select.cc:1832
        curr_table = (JOIN_TAB *) 0x24
        used_tables = 147409216
        sort_table_cond = (class Item *) 0xc0000000
        columns_list = <value optimized out>
        tmp_error = <value optimized out>
        _db_func_ = 0x8c8e468 "(ÌU\bè5Å\b\2146Å\b"
        _db_file_ = 0x8c5394c "pÈÈ\b@ÖÈ\b\a"
        _db_level_ = 0
        _db_framep_ = (char **) 0xb3000454
        curr_join = (JOIN *) 0x8c92678
        curr_all_fields = (List<Item> *) 0x8c933c8
        curr_fields_list = (List<Item> *) 0x8c933ec
        curr_tmp_table = (TABLE *) 0x8cb61d0
        __PRETTY_FUNCTION__ = "void JOIN::exec()"
#3  0x0824d8a5 in mysql_select (thd=0x8c535e8, rref_pointer_array=0x8c539dc, tables=0x8c8d678, wild_num=0, fields=@0x8c5394c, conds=0x8c8dd40, og_num=6, 
    order=0x8c8dff8, group=0x8c8df00, having=0x0, proc_param=0x0, select_options=2156153344, result=0x8c8e468, unit=0x8c5368c, select_lex=0x8c538c0)
    at sql_select.cc:2045
        err = false
---Type <return> to continue, or q <return> to quit---
        free_join = true
        _db_func_ = 0xb30004d0 "è5Å\bÜ9Å\bxÖÈ\b"
        _db_file_ = 0xb300052b "³xÖÈ\bè5Å\bhäÈ\bÈ\f"
        _db_level_ = 3003122896
        _db_framep_ = (char **) 0xb30004d4
        join = (JOIN *) 0x8c92678
#4  0x0824dda8 in handle_select (thd=0x8c535e8, lex=0x8c53628, result=0x8c8e468, setup_tables_done_option=0) at sql_select.cc:256
        unit = <value optimized out>
        res = <value optimized out>
        select_lex = (SELECT_LEX *) 0x8c538c0
        _db_func_ = 0xb3000538 "È\f"
        _db_file_ = 0x0
        _db_level_ = 147383400
        _db_framep_ = (char **) 0xb7f183f5
#5  0x081f9dbf in mysql_execute_command (thd=0x8c535e8) at sql_parse.cc:2615
        xs = <value optimized out>
        res = <value optimized out>
        need_start_waiting = <value optimized out>
        result = <value optimized out>
        lex = (LEX *) 0x8c53628
        select_lex = (SELECT_LEX *) 0x8c538c0
        first_table = (TABLE_LIST *) 0x8c8d678
        all_tables = (TABLE_LIST *) 0x8c8d678
        unit = (SELECT_LEX_UNIT *) 0x8c5368c
        _db_func_ = 0x0
        _db_file_ = 0x0
        _db_level_ = 0
        _db_framep_ = (char **) 0x0
        __PRETTY_FUNCTION__ = "bool mysql_execute_command(THD*)"
#6  0x081fad53 in mysql_parse (thd=0x8c535e8, 
    inBuf=0x8c8c628 "select  bar.s,\n        group_concat(bar.gt),\n        min(bar.dt) as minDT,\n        min(foo.p) as minP,\n        sum(bar.nr) as sumNR,\n        sum(bar.n) as sumN,\n        min(bar.gi) as minGI\nfrom    fo"..., length=314) at sql_parse.cc:5897
        lex = (LEX *) 0x8c53628
        _db_func_ = 0xff0a0000 <Address 0xff0a0000 out of bounds>
        _db_file_ = 0x73650074 <Address 0x73650074 out of bounds>
        _db_level_ = 0
        _db_framep_ = (char **) 0xce
        __PRETTY_FUNCTION__ = "void mysql_parse(THD*, char*, uint)"
---Type <return> to continue, or q <return> to quit---
#7  0x081fb9f8 in dispatch_command (command=COM_QUERY, thd=0x8c535e8, 
    packet=0x8c845c9 "select  bar.s,\n        group_concat(bar.gt),\n        min(bar.dt) as minDT,\n        min(foo.p) as minP,\n        sum(bar.nr) as sumNR,\n        sum(bar.n) as sumN,\n        min(bar.gi) as minGI\nfrom    fo"..., packet_length=315) at sql_parse.cc:1774
        packet_end = 0x8c8c762 ""
        net = (NET *) 0x8c53e18
        error = <value optimized out>
        _db_func_ = 0x1 <Address 0x1 out of bounds>
        _db_file_ = 0xb3001078 "¨\020"
        _db_level_ = 3085575516
        _db_framep_ = (char **) 0x3
#8  0x081fcd26 in do_command (thd=0x8c535e8) at sql_parse.cc:1556
        packet = 0x8c845c8 "\003select  bar.s,\n        group_concat(bar.gt),\n        min(bar.dt) as minDT,\n        min(foo.p) as minP,\n        sum(bar.nr) as sumNR,\n        sum(bar.n) as sumN,\n        min(bar.gi) as minGI\nfrom    f"...
        old_timeout = 30
        packet_length = 315
        net = (NET *) 0x8c53e18
        command = COM_QUERY
        _db_func_ = 0x1000 <Address 0x1000 out of bounds>
        _db_file_ = 0x2000 <Address 0x2000 out of bounds>
        _db_level_ = 147146800
        _db_framep_ = (char **) 0x81c962f
#9  0x081fdb6d in handle_one_connection (arg=0x8c535e8) at sql_parse.cc:1187
        error = <value optimized out>
        net = (NET *) 0x8c53e18
        sctx = (Security_context *) 0x8c545f8
        thd = (class THD *) 0x8c535e8
        set = {__val = {0 <repeats 32 times>}}
#10 0xb7f14504 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
No symbol table info available.
#11 0xb7e3e51e in clone () from /lib/tls/i686/cmov/libc.so.6
No symbol table info available.
[24 Jan 2007 1:58] MySQL Verification Team
Thank you for the bug report.
[24 Jan 2007 16:15] Heikki Tuuri
The stack trace suggests this is a MySQL server bug.
[26 Jan 2007 19:34] Pete Harlan
I neglected to mention that at the time of one of the crashes I got this message on the console.  I didn't see it again.  I don't know where the address points, and this was with a non-debug version of mysql that I have since written over, but perhaps the message is interesting.

*** glibc detected *** double free or corruption (out): 0x00000000022ded00 ***
[30 Jan 2007 1:34] Igor Babaev
This problem could be demonstrated for MyISAM as well:

mysql> CREATE TABLE foo (
    ->   id int(11) NOT NULL,
    ->   gi int(11) NOT NULL,
    ->   gt varchar(5) NOT NULL,
    ->   p int(11) NOT NULL,
    ->   PRIMARY KEY  (gt,gi,id)
    -> );
Query OK, 0 rows affected (0.47 sec)

mysql> INSERT INTO foo VALUES (170,13674,'c',2);
Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO foo VALUES (170,13674,'d',2);
Query OK, 1 row affected (0.00 sec)

mysql> CREATE TABLE bar (
    ->   gt varchar(5) NOT NULL,
    ->   gi int(11) NOT NULL,
    ->   s varchar(10) NOT NULL,
    ->   nr int(11) NOT NULL,
    ->   n int(11) NOT NULL,
    ->   dt int(11) NOT NULL,
    ->   PRIMARY KEY  (gt,gi)
    -> );
Query OK, 0 rows affected (0.47 sec)

mysql> INSERT INTO bar VALUES ('c',13674,'13834a0730',0,0,2007);
Query OK, 1 row affected (0.01 sec)

mysql> INSERT INTO bar VALUES ('d',13674,'13834a0730',0,0,2007);
Query OK, 1 row affected (0.00 sec)

mysql> select  bar.s,
    ->         group_concat(bar.gt),
    ->         min(bar.dt) as minDT,
    ->         min(foo.p) as minP,
    ->         sum(bar.nr) as sumNR,
    ->         sum(bar.n) as sumN,
    ->         min(bar.gi) as minGI
    -> from    foo join bar using (gt, gi)
    -> where   foo.id  = 170
    -> group by bar.s
    -> order by minDT, minP desc, sumNR, sumN, bar.gt desc;
ERROR 2013 (HY000): Lost connection to MySQL server during query

Yet this crash happened by the same reason as the crash reported as bug #25172.
After the patch fixing the problem of #25172 we have:

mysql> select  bar.s,
    ->         group_concat(bar.gt),
    ->         min(bar.dt) as minDT,
    ->         min(foo.p) as minP,
    ->         sum(bar.nr) as sumNR,
    ->         sum(bar.n) as sumN,
    ->         min(bar.gi) as minGI
    -> from    foo join bar using (gt, gi)
    -> where   foo.id  = 170
    -> group by bar.s
    -> order by minDT, minP desc, sumNR, sumN, bar.gt desc;
+------------+----------------------+-------+------+-------+------+-------+
| s          | group_concat(bar.gt) | minDT | minP | sumNR | sumN | minGI |
+------------+----------------------+-------+------+-------+------+-------+
| 13834a0730 | d,c                  |  2007 |    2 |     0 |    0 | 13674 |
+------------+----------------------+-------+------+-------+------+-------+
1 row in set (0.00 sec)

So this bug #25804 can be considered a duplicate of bug #25172 which is already
fixed in the main trees for 5.0 and 5.1.