MySQL Bugs: #26625: crash in range optimizer (out of mem) Bug #26624

Bug #26625	crash in range optimizer (out of mem) Bug #26624
Submitted:	25 Feb 2007 21:49	Modified:	31 Mar 2007 23:07
Reporter:	Martin Friebe (Gold Quality Contributor) (OCA)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	5.2.4BK, 5.1.17BK, 5.0.38BK 4.1.23BK	OS:	Any (*)
Assigned to:	Sergey Petrunya	CPU Architecture:	Any
Tags:	crash, Memory, Optimizer, qc, range

Description:
The query below does crash the range optimizer, as it runs out of memory.

Even if bug #26624 (which causes this particular query to use so much memory) is fixed, it will still be possible to feed mysql queries that use a lot of memory inside the range optimizer; while at the same time the overall amount of memory could be running low through other queries.

Mysql should report an error instead of crashing.

How to repeat:
drop table if exists xl1;
create table xl1 (
c1 char(10), c2 char(10), c3 char(10), c4 char(10),
c5 char(10), c6 char(10), c7 char(10), c8 char(10),
c9 char(10), c10 char(10),c11 char(10), c12 char(10),
c13 char(10),c14 char(10), c15  char(10), c16  char(10),
index( c1, c2, c3, c4, c5, c6, c7, c8, c9, c10,
 c11, c12,c13,c14,c15,c16 )
);

insert into xl1 (c1) select '1';
insert into xl1 (c1) select '1';
insert into xl1 (c1) select '1';
insert into xl1 (c1) select '1';

select * from xl1 where
 c1 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c2 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c3 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c4 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c5 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c6 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c7 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c8 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c9 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c10 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c11 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c12 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c13 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c14 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c15 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
 and c16 in ("abcdefgh", "123456789", "qwertyuio", "asddfgh")
;

Suggested fix:
While within most of the code, object creation is checked for "out of mem" like the following:
    if (!(tmp= new SEL_ARG(field,part, min_value,max_value,
			   min_flag, max_flag, maybe_flag)))
      return 0;					// OOM

this does not apply to most of the constructors:

SEL_ARG::SEL_ARG(Field *f,const char *min_value_arg,const char *max_value_arg)
  :min_flag(0), max_flag(0), maybe_flag(0), maybe_null(f->real_maybe_null()),
  elements(1), use_count(1), field(f), min_value((char*) min_value_arg),
  max_value((char*) max_value_arg), next(0),prev(0),
  next_key_part(0),color(BLACK),type(KEY_RANGE)
{
    left=right= &null_element;
}

The above constructor will be executed, even if memory allocation failed. This is if the overloaded "new" operator from the parent class Sql_alloc returns a pointer to address 0.

At least "gcc (GCC) 3.4.2 [FreeBSD]" does not seem to care about that 0 pointer.

  if(this)
    left=right= &null_element;

is also not sufficent, as the initialization of the other fields will still happen.

I have tried to fix the constructors for SEL_ARG, but there must be more locations were out of mem is not handled. Yet fixing the SEL_ARG constructors did change (repeatable) the location of the crash, if looking at a stack trace

experimental, changing constructors to check for out of mem. (this does not completly fix the issue)

Attachment: sel_arg_nomem.patch (text/x-patch), 4.68 KiB.

5.0.38BK crash

Attachment: bug26625_stack_5.0.38.txt (text/plain), 5.08 KiB.

thanks for a bug report, verified as described

Martin, 

Thanks for figuring out the problem with constructors. However, if we follow the direction taken in your patch, we will have to make such changes all over the code. Also, it seems that this solution won't work for classes with virtual function.

Ok, the problems with NULL pointer dereferencing in constructor can be solved by declaring operator new as no-throw (thanks to Dmitry Lenev for pointing this out):

===== sql_list.h 1.32 vs edited =====
--- 1.32/sql/sql_list.h 2007-03-26 12:54:12 +04:00
+++ edited/sql_list.h   2007-03-13 07:32:27 +03:00
@@ -30,7 +30,7 @@
 class Sql_alloc
 {
 public:
-  static void *operator new(size_t size)
+  static void *operator new(size_t size) throw ()
   {
     return (void*) sql_alloc((uint) size);
   }
@@ -38,7 +38,7 @@
   {
     return (void*) sql_alloc((uint) size);
   }
-  static void *operator new(size_t size, MEM_ROOT *mem_root)
+  static void *operator new(size_t size, MEM_ROOT *mem_root) throw ()
   { return (void*) alloc_root(mem_root, (uint) size); }
   static void operator delete(void *ptr, size_t size) { TRASH(ptr, size); }
   static void operator delete(void *ptr, MEM_ROOT *mem_root)

Besides the above fix, a proper fix for this bug will either
1. Make range optimizer to never require combinatorial amounts of memory, or

2. (if we can't do #1) Limit the amount of memory that can be consumed by the range optimizer.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/23166

ChangeSet@1.2626, 2007-03-28 18:38:42+04:00, sergefp@mysql.com +1 -0
  BUG#26625: crash in range optimizer (out of mem) 
  - Define Sql_alloc::operator new() as thow() so that C++ compiler
    handles NULL return values
  (there is no testcase as there is no portable way to set limit on the 
  amount of memory that a process can allocate)

Pushed into 5.1.18-beta

Pushed into 5.0.40

Pushed into 4.1.23

Noted in 4.1.23, 5.0.40, 5.1.18 changelogs.

The range optimizer could cause the server to run out of memory.