Bug #48060 Memory leak - Item::val_bool() (item.cc:184) from optimizer_subquery grammar
Submitted: 14 Oct 2009 21:20 Modified: 17 Nov 2009 23:17
Reporter: Patrick Crews Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Optimizer Severity:S3 (Non-critical)
Version:5.5, 6.0-codebase OS:Any
Assigned to: Tor Didriksen CPU Architecture:Any
Tags: memory leak, valgrind
Triage: Triaged: D2 (Serious)

[14 Oct 2009 21:20] Patrick Crews
Description:
Memory leak detected during a run of the optimizer_subquery.yy grammar:

20,405,852 (18,264,214 direct, 2,141,638 indirect) bytes in 5,364 blocks are definitely lost in loss record 6 of 8
10115:==2502==    at 0x4026FDE: malloc (vg_replace_malloc.c:207)
10116:==2502==    by 0x877E95B: my_malloc (my_malloc.c:37)
10117:==2502==    by 0x877EB75: my_strdup (my_malloc.c:85)
10118:==2502==    by 0x877DD7A: open_cached_file (mf_cache.c:65)
10119:==2502==    by 0x83D9FC9: Unique::Unique(int (*)(void*, void const*, void const*), void*, unsigned int, unsigned long long) (uniques.cc:70)
10120:==2502==    by 0x8245CEC: Aggregator_distinct::setup(THD*) (item_sum.cc:890)
10121:==2502==    by 0x83B0628: Item_sum::aggregator_setup(THD*) (item_sum.h:480)
10122:==2502==    by 0x837D71F: setup_sum_funcs(THD*, Item_sum**) (sql_select.cc:20728)
10123:==2502==    by 0x83AF5DD: JOIN::exec() (sql_select.cc:2743)
10124:==2502==    by 0x82A6A5C: subselect_single_select_engine::exec() (item_subselect.cc:2310)
10125:==2502==    by 0x82AB5CE: Item_subselect::exec() (item_subselect.cc:283)
10126:==2502==    by 0x82AB75E: Item_in_subselect::exec() (item_subselect.cc:333)
10127:==2502==    by 0x82A370D: Item_in_subselect::val_bool() (item_subselect.cc:923)
10128:==2502==    by 0x82371E3: Item::val_bool_result() (item.h:760)
10129:==2502==    by 0x82698EC: Item_in_optimizer::val_int() (item_cmpfunc.cc:1611)
10130:==2502==    by 0x8225D4B: Item::val_bool() (item.cc:184)

How to repeat:
Use the RQG:

perl ./runall.pl --threads=1 --basedir=<path>/mysql-6.0 --validator=MarkErrorLog --valgrind --views  --queries=10000 --mysqld=--init-file=<path>/6.sql --grammar=conf/optimizer_subquery.yy --reporter=Shutdown

init-file contents
SET GLOBAL OPTIMIZER_SWITCH = 'materialization=off,semijoin=off,loosescan=off,firstmatch=off';

SET GLOBAL optimizer_use_mrr = 'disable';

SET GLOBAL debug = '+d,optimizer_no_icp';

SET GLOBAL engine_condition_pushdown = 1;

SET GLOBAL join_cache_level = 1;
[19 Oct 2009 8:49] Tor Didriksen
The same memory leak can be observed in about a dozen of the tests in the main
test suite.

Here's a simple example to re-create the memory leak

create table t1 (a int, b int);
create table t2 (a int, b int);
create table t3 (a int, b int);
insert into t1 values (0,100),(1,2), (1,3), (2,2), (2,7), (2,-1), (3,10);
insert into t2 values (0,0), (1,1), (2,1), (3,1), (4,1);
insert into t3 values (3,3), (2,2), (1,1);
select a,(select count(distinct t1.b) as sum from t1,t2 where t1.a=t2.a and t2.b > 0 and t1.a <= t3.b group by t1.a order by sum limit 1) from t3;
drop table t1,t2,t3;

The problem seems to be that we call prepare_sum_aggregators()
multiple times. Each invocation of Item_sum::set_aggregator()
will do
    aggr= new Aggregator_distinct(this);
and we will leak the previous contents of aggr.

Changing set_aggregator() as shown below, removes all leaks of this sort in the main test suite.

int Item_sum::set_aggregator(Aggregator::Aggregator_type aggregator)
{
  DBUG_PRINT("info", ("set_Aggregator this %p old %p", this, aggr));
  if (aggr)
  {
    DBUG_ASSERT(aggregator == aggr->Aggrtype());
    return FALSE;
  }
  switch (aggregator)
  {
  case Aggregator::DISTINCT_AGGREGATOR:
    aggr= new Aggregator_distinct(this);
    break;

  case Aggregator::SIMPLE_AGGREGATOR:
    aggr= new Aggregator_simple(this);
    break;
  };
  DBUG_PRINT("info", ("set_Aggregator this %p new %p", this, aggr));
  return aggr ? FALSE : TRUE;
}
[19 Oct 2009 10:30] Tor Didriksen
Same leak in mysql-next-mr
../sql/mysqld  Ver 5.5.0-beta-valgrind-max-debug for pc-linux-gnu on i686 (Source distribution)

==25348== 9,166 (6,822 direct, 2,344 indirect) bytes in 12 blocks are definitely lost in loss record 8 of 9
==25348==    at 0x4006AEE: malloc (vg_replace_malloc.c:207)
==25348==    by 0x86CC09C: my_malloc (my_malloc.c:34)
==25348==    by 0x86CCA7E: alloc_root (my_alloc.c:158)
==25348==    by 0x86CCBCA: multi_alloc_root (my_alloc.c:264)
==25348==    by 0x83092F9: create_tmp_table(THD*, TMP_TABLE_PARAM*, List<Item>&, st_order*, bool, bool, unsigned long long, unsigned long long, char*) (
sql_select.cc:9859)
==25348==    by 0x81D4DBE: Aggregator_distinct::setup(THD*) (item_sum.cc:762)
==25348==    by 0x8322D6C: Item_sum::aggregator_setup(THD*) (item_sum.h:482)
==25348==    by 0x82F8D73: _ZL15setup_sum_funcsP3THDPP8Item_sum (sql_select.cc:15437)
==25348==    by 0x8321714: JOIN::exec() (sql_select.cc:1979)
==25348==    by 0x8231C11: subselect_single_select_engine::exec() (item_subselect.cc:1960)
==25348==    by 0x8236B54: Item_subselect::exec() (item_subselect.cc:261)
==25348==    by 0x8230118: Item_singlerow_subselect::val_int() (item_subselect.cc:565)
==25348==    by 0x81B3639: Item::send(Protocol*, String*) (item.cc:5559)
==25348==    by 0x8269245: select_send::send_data(List<Item>&) (sql_class.cc:1689)
==25348==    by 0x82F9801: _ZL8end_sendP4JOINP13st_join_tableb (sql_select.cc:12091)
==25348==    by 0x83009F2: _ZL20evaluate_join_recordP4JOINP13st_join_tablei (sql_select.cc:11347)
[26 Oct 2009 15:22] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/88188

2921 Tor Didriksen	2009-10-26
      Bug#48060 Memory leak - Item::val_bool() (item.cc:184) from optimizer_subquery grammar
      
      Item_sum::set_aggregator() may be called multiple times during query preparation.
      On subsequent calls: verify that the aggregator type is the same,
      and re-use the existing Aggregator.
     @ sql/item_sum.cc
        In Item_sum::set_aggregator(): re-use existing Aggregator if already set.
        
        Remove some friend declarations, add some accessor functions.
        Cleanup some DBUG_ENTER and DBUG_RETURN code.
     @ sql/item_sum.h
        Make some member fields private, add accessors instead.
        Remove some un-necessary friend declarations.
        Remove some default arguments from constructors.
     @ sql/opt_sum.cc
        Use accessor functions in Item_sum.
     @ sql/sql_select.cc
        Fix mis-spelled DBUG_ENTER text.
        Use accessor functions in Item_sum.
     @ sql/sql_yacc.yy
        Use explicit true/false rather than default arguments when constructing
        Item_sum_xxx objects.
[27 Oct 2009 15:38] Georgi Kodinov
OK to push with one small note  (see below) :  

@@ -855,23 +859,23 @@ bool Aggregator_distinct::setup(THD *thd
       mem_root.
     */
     if (field_list.push_back(&field_def))
-      return TRUE;
+      DBUG_RETURN(TRUE);
 
     item_sum->null_value= item_sum->maybe_null= 1;
     item_sum->quick_group= 0;
 
     DBUG_ASSERT(item_sum->get_arg(0)->fixed);
 
-    arg = item_sum->get_arg(0);
+    Item *arg= item_sum->get_arg(0);

Joro:^^^^^^
Joro:Please remove the local declaration. I don't think it's needed
[28 Oct 2009 7:04] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/88438

2926 Tor Didriksen	2009-10-26
      Bug#48060 Memory leak - Item::val_bool() (item.cc:184) from optimizer_subquery grammar
      
      Item_sum::set_aggregator() may be called multiple times during query preparation.
      On subsequent calls: verify that the aggregator type is the same,
      and re-use the existing Aggregator.
     @ sql/item_sum.cc
        In Item_sum::set_aggregator(): re-use existing Aggregator if already set.
        
        Remove some friend declarations, add some accessor functions.
        Cleanup some DBUG_ENTER and DBUG_RETURN code.
     @ sql/item_sum.h
        Make some member fields private, add accessors instead.
        Remove some un-necessary friend declarations.
        Remove some default arguments from constructors.
     @ sql/opt_sum.cc
        Use accessor functions in Item_sum.
     @ sql/sql_select.cc
        Fix mis-spelled DBUG_ENTER text.
        Use accessor functions in Item_sum.
     @ sql/sql_yacc.yy
        Use explicit true/false rather than default arguments when constructing
        Item_sum_xxx objects.
[28 Oct 2009 10:07] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/88465

2927 Tor Didriksen	2009-10-28
      Bug#48060 Memory leak - Item::val_bool() (item.cc:184) from optimizer_subquery grammar
      
      Item_sum::set_aggregator() may be called multiple times during query preparation.
      On subsequent calls: verify that the aggregator type is the same,
      and re-use the existing Aggregator.
     @ sql/item_sum.cc
        In Item_sum::set_aggregator(): re-use existing Aggregator if already set.
        
        Remove some friend declarations, add some accessor functions.
        Cleanup some DBUG_ENTER and DBUG_RETURN code.
     @ sql/item_sum.h
        Make some member fields private, add accessors instead.
        Remove some un-necessary friend declarations.
        Remove some default arguments from constructors.
     @ sql/opt_sum.cc
        Use accessor functions in Item_sum.
     @ sql/sql_select.cc
        Fix mis-spelled DBUG_ENTER text.
        Use accessor functions in Item_sum.
     @ sql/sql_yacc.yy
        Use explicit true/false rather than default arguments when constructing
        Item_sum_xxx objects.
[28 Oct 2009 11:16] Tor Didriksen
pushed to: bzr+ssh://bk-internal.mysql.com/bzrroot/server/mysql-next-mr-bugfixing
[28 Oct 2009 12:49] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/88478

3683 Tor Didriksen	2009-10-28 [merge]
      Merge fix for Bug#48060 from my:next-mr-bugfixing
[28 Oct 2009 13:30] Tor Didriksen
pushed to
bzr+ssh://bk-internal.mysql.com/bzrroot/server/mysql-6.0-codebase-bugfixing/
[31 Oct 2009 8:19] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091031081410-qkxmjsdzjmj840aq) (version source revid:tor.didriksen@sun.com-20091028124831-l34xkirry2fu5gtd) (merge vers: 6.0.14-alpha) (pib:13)
[12 Nov 2009 8:17] Bugs System
Pushed into 5.5.0-beta (revid:alik@sun.com-20091110093229-0bh5hix780cyeicl) (version source revid:mikael@mysql.com-20091102100915-a2nbfxaqprpgptfw) (merge vers: 5.5.0-beta) (pib:13)
[17 Nov 2009 23:17] Paul Dubois
Noted in 5.5.0, 6.0.14 changelogs.

The subquery optimizer had a memory leak.
[29 Mar 2010 8:41] Tor Didriksen
See Bug#48920 and Bug#52123