Bug #44886 SIGSEGV in test_if_skip_sort_order() - uninitialized variable used as subscript

Description:
crash with SIGSEGV in sql_select.cc::test_if_skip_sort_order() 

At line 13129
      rec_per_key= keyinfo->rec_per_key[used_key_parts-1];

used_key_parts can be used uninitialized, thus crashing or reading random memory. 

used_key_parts is initialized by test_if_order_by_key(), but this one has many exit points that won't assign *used_key_parts. The test case demonstrates it quite clearly.

How to repeat:
100% reproducible test case:

drop table IF EXISTS FOO;
CREATE TABLE `FOO` (
  `FOO_ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `A` int(20) DEFAULT NULL,
  PRIMARY KEY (`FOO_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
insert into FOO VALUES(NULL,1);
insert into FOO VALUES(NULL,2);

drop table IF EXISTS BAR;
CREATE TABLE `BAR` (
  `BAR_ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `FOO_ID` int(10) unsigned NOT NULL DEFAULT '0',
  `TYPE` smallint(5) unsigned NOT NULL DEFAULT '1',
  `STATUS` bigint(20) unsigned DEFAULT '0',
  PRIMARY KEY (`BAR_ID`),
  KEY `IDX_FOO` (`FOO_ID`,`TYPE`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
insert into BAR VALUES(NULL,1,1,0);
insert into BAR VALUES(NULL,2,1,0);

drop table IF EXISTS BAZ;
CREATE TABLE `BAZ` (
  `BAZ_ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `FOO_ID` int(10) unsigned NOT NULL DEFAULT '0',
  `BAR_ID` int(10) unsigned NOT NULL,
  `BAZ_TAG` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`BAZ_ID`),
  UNIQUE KEY `UDX_FOO_BAR` (`FOO_ID`,`BAR_ID`),
  KEY `IDX_BAZTAG` (`BAZ_TAG`)
) ENGINE=InnoDB CHARSET=latin1;
insert into BAZ values(NULL,1,1,'foo@bar.com');
insert into BAZ values(NULL,2,1,'foo@bar.com');

DROP TABLE IF EXISTS FOO_BAR;
CREATE TABLE `FOO_BAR` (
  `FOO_ID` int(10) unsigned DEFAULT '0', 
  `BAR_ID` int(10) unsigned DEFAULT '0'
) ENGINE=MyISAM CHARSET=latin1; # Does not crash with ENGINE=InnoDB
INSERT INTO FOO_BAR VALUES (1, 1);

SELECT  FOO.FOO_ID
FROM    FOO_BAR
        INNER JOIN FOO ON (FOO.FOO_ID = FOO_BAR.FOO_ID)
        INNER JOIN BAR ON (BAR.FOO_ID = FOO.FOO_ID AND (BAR.BAR_ID = FOO_BAR.BAR_ID OR BAR.STATUS & 1))
        LEFT JOIN BAZ ON (BAZ.BAR_ID = BAR.BAR_ID  AND BAZ_TAG = 'foo@bar.com')
WHERE   BAZ_TAG = 'foo@bar.com'
GROUP BY BAR.BAR_ID;

Suggested fix:
--- mysql-5.1.34/sql/sql_select.cc.ori	2009-05-14 09:23:16.000000000 -0400
+++ mysql-5.1.34/sql/sql_select.cc	2009-05-14 09:23:16.000000000 -0400
@@ -12912,7 +12912,7 @@
   int ref_key;
   uint ref_key_parts;
   int order_direction;
-  uint used_key_parts;
+  uint used_key_parts= 0;
   TABLE *table=tab->table;
   SQL_SELECT *select=tab->select;
   key_map usable_keys;
@@ -13124,7 +13124,7 @@
           KEY *keyinfo= tab->table->key_info+nr;
           if (select_limit == HA_POS_ERROR)
             select_limit= table_records;
-          if (group)
+          if (group && used_key_parts)
           {
             rec_per_key= keyinfo->rec_per_key[used_key_parts-1];
             set_if_bigger(rec_per_key, 1);

Thank you for the report.

Backtrace in my environment:

Thread 1 (process 19545):
#0  0x002ce402 in __kernel_vsyscall ()
#1  0x0046264f in pthread_kill () from /lib/libpthread.so.0
#2  0x085a7585 in my_write_core (sig=11) at stacktrace.c:310
#3  0x0824d392 in handle_segfault (sig=11) at mysqld.cc:2536
#4  <signal handler called>
#5  0x082df93c in test_if_skip_sort_order (tab=0x97124b0, order=0x970ed18, select_limit=2, no_changes=false, map=0x96ffb0c) at sql_select.cc:13141
#6  0x082e9e76 in JOIN::optimize (this=0x970edc8) at sql_select.cc:1360
#7  0x082efda1 in mysql_select (thd=0x96a8d38, rref_pointer_array=0x96aa1d0, tables=0x9723cd8, wild_num=0, fields=@0x96aa16c, conds=0x970eb88, og_num=1, order=0x0, group=0x970ed18, having=0x0, 
    proc_param=0x0, select_options=2147764736, result=0x970edb0, unit=0x96a9e68, select_lex=0x96aa0d8) at sql_select.cc:2364
#8  0x082f017f in handle_select (thd=0x96a8d38, lex=0x96a9e0c, result=0x970edb0, setup_tables_done_option=0) at sql_select.cc:268
#9  0x0825c757 in execute_sqlcom_select (thd=0x96a8d38, all_tables=0x9723cd8) at sql_parse.cc:5009
#10 0x08262a87 in mysql_execute_command (thd=0x96a8d38) at sql_parse.cc:2211
#11 0x0826c258 in mysql_parse (thd=0x96a8d38, 
    inBuf=0x9723960 "SELECT  FOO.FOO_ID\nFROM    FOO_BAR\nINNER JOIN FOO ON (FOO.FOO_ID = FOO_BAR.FOO_ID)\nINNER JOIN BAR ON (BAR.FOO_ID = FOO.FOO_ID AND (BAR.BAR_ID = FOO_BAR.BAR_ID OR\nBAR.STATUS & 1))\nLEFT JOIN BAZ ON (BAZ"..., length=302, found_semicolon=0xb2b992fc) at sql_parse.cc:5929
#12 0x0826ce94 in dispatch_command (command=COM_QUERY, thd=0x96a8d38, 
    packet=0x96eddb1 "SELECT  FOO.FOO_ID\nFROM    FOO_BAR\nINNER JOIN FOO ON (FOO.FOO_ID = FOO_BAR.FOO_ID)\nINNER JOIN BAR ON (BAR.FOO_ID = FOO.FOO_ID AND (BAR.BAR_ID = FOO_BAR.BAR_ID OR\nBAR.STATUS & 1))\nLEFT JOIN BAZ ON (BAZ"..., packet_length=302) at sql_parse.cc:1216
#13 0x0826e0b2 in do_command (thd=0x96a8d38) at sql_parse.cc:857
#14 0x0825a831 in handle_one_connection (arg=0x96a8d38) at sql_connect.cc:1115
#15 0x0045fbd4 in start_thread () from /lib/libpthread.so.0
#16 0x003b74fe in clone () from /lib/libc.so.6

There is similar bug #37868

Bug does not exist in versions 4.1 and 5.0

This bug was introduced by the small part of a bug #31001 fix:

@@ -12353,6 +12353,12 @@ static int test_if_order_by_key(ORDER *o

         for (; const_key_parts & 1 ; const_key_parts>>= 1)
           key_part++;
+        /*
+         The primary and secondary key parts were all const (i.e. there's
+         one row).  The sorting doesn't matter.
+        */
+        if (key_part == key_part_end && reverse == 0)
+          DBUG_RETURN(1);
       }
       else
         DBUG_RETURN(0);

This new code fragment returns value of 1 ("key is ok"), but it
doesn't update the used_key_parts output parameter of the test_if_order_by_key
function. That causes a crash.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/75737

2929 Gleb Shchepa	2009-06-06
      Bug #44886: SIGSEGV in test_if_skip_sort_order() -
                  uninitialized variable used as subscript
      
      Grouping select from a "constant" InnoDB table (a table
      of a single row) joined with other tables caused a crash.
     @ mysql-test/r/innodb_mysql.result
        Added test case for bug bug #44886.
     @ mysql-test/t/innodb_mysql.test
        Added test case for bug bug #44886.
     @ sql/sql_select.cc
        Bug #44886: SIGSEGV in test_if_skip_sort_order() -
                    uninitialized variable used as subscript
        
        1. The test_if_order_by_key function returned unitialized
           used_key_parts parameter in case of a "constant" InnoDB
           table. Calling function uses this parameter values as
           an array index, thus sometimes it caused a crash.
           The test_if_order_by_key function has been modified
           to set used_key_parts to 0 (no need for ordering).
        
        2. The test_if_skip_sort_order function has been
           modified to accept zero used_key_parts value and
           to prevent an array access by negative index.

Pushed into 5.1.36 (revid:joro@sun.com-20090616102155-3zhezogudt4uxdyn) (version source revid:gshchepa@mysql.com-20090607204053-0kai6dlgdwdbohlv) (merge vers: 5.1.36) (pib:6)

Pushed into 5.4.4-alpha (revid:alik@sun.com-20090616183122-chjzbaa30qopdra9) (version source revid:gshchepa@mysql.com-20090607205014-t64c0ktv7i58p62d) (merge vers: 6.0.12-alpha) (pib:11)

Noted in 5.1.36, 5.4.4 changelogs.

GROUP BY on a constant (single-row) InnoDB table joined to other
tables caused a server crash.

Noted in 5.4.2 changelog because next 5.4 version will be 5.4.2 and not 5.4.4.

Ignore previous comment about 5.4.2.

Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)

Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)

Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

The 5.4 fix has been pushed to 5.4.2.