MySQL Bugs: #33675: crash with subquery and filesort?

Bug #33675	crash with subquery and filesort?
Submitted:	3 Jan 2008 20:05	Modified:	25 Jan 2008 19:24
Reporter:	Shane Bester (Platinum Quality Contributor)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Optimizer	Severity:	S1 (Critical)
Version:	5.0.54, 5.1.23	OS:	Any
Assigned to:	Evgeny Potemkin	CPU Architecture:	Any
Tags:	crash, filesort, sort_buffer_size, subquery

Description:
After filesort is done by subquery, a crash occurs with stacktrace:

mysqld-nt.exe!memset
mysqld-nt.exe!Field_string::unpack
mysqld-nt.exe!unpack_addon_fields
mysqld-nt.exe!rr_unpack_from_tempfile
mysqld-nt.exe!join_init_read_record
mysqld-nt.exe!sub_select
mysqld-nt.exe!do_select
mysqld-nt.exe!JOIN::exec
mysqld-nt.exe!subselect_single_select_engine::exec
mysqld-nt.exe!Item_subselect::exec
mysqld-nt.exe!Item_singlerow_subselect::val_int
mysqld-nt.exe!Arg_comparator::compare_int_unsigned_signed
mysqld-nt.exe!Item_func_eq::val_int
mysqld-nt.exe!evaluate_join_record
mysqld-nt.exe!sub_select
mysqld-nt.exe!evaluate_join_record
mysqld-nt.exe!sub_select
mysqld-nt.exe!do_select
mysqld-nt.exe!JOIN::exec
mysqld-nt.exe!mysql_select
mysqld-nt.exe!handle_select
mysqld-nt.exe!mysql_execute_command
mysqld-nt.exe!mysql_parse
mysqld-nt.exe!dispatch_command
mysqld-nt.exe!do_command
mysqld-nt.exe!handle_one_connection
mysqld-nt.exe!pthread_start
mysqld-nt.exe!_callthreadstart
mysqld-nt.exe!_threadstart
kernel32.dll!BaseThreadStart

How to repeat:
see attached private testcase.  I managed to get the size of a crashing testcase down to 400kb on 5.0.50, but then 5.0.54 didn't crash.  So, I am using the larger more consistent testcase that crashes all reported versions (16M)

seems the crash is related to the value of sort_buffer_size

full stack with variables..

Attachment: bug33675_full_stack_trace.txt (text/plain), 3.91 KiB.

ok, will upload larger testcase shortly.
here I tried 5.0.54 on linux:

0x818c729 handle_segfault + 417
0x40125ac7 _end + 934427351
0x8233895 unpack_addon_fields(st_sort_addon_field*, char*) + 105
0x823154c rr_unpack_from_tempfile(st_read_record*) + 68
0x81e6ced join_init_read_record(st_join_table*) + 61
0x81d6dbb sub_select(JOIN*, st_join_table*, bool) + 251
0x81ddb34 do_select(JOIN*, List<Item>*, st_table*, Procedure*) + 260
0x81d2d3a JOIN::exec() + 4426
0x8165ea0 subselect_single_select_engine::exec() + 280
0x8161cd0 Item_subselect::exec() + 44
0x81624c3 Item_singlerow_subselect::val_int() + 19
0x813e7b6 Arg_comparator::compare_int_unsigned_signed() + 74
0x813f16d Item_func_eq::val_int() + 45
0x81ddde7 evaluate_join_record(JOIN*, st_join_table*, int, char*) + 163
0x81d6dc6 sub_select(JOIN*, st_join_table*, bool) + 262
0x81dde81 evaluate_join_record(JOIN*, st_join_table*, int, char*) + 317
0x81d6dc6 sub_select(JOIN*, st_join_table*, bool) + 262
0x81ddb34 do_select(JOIN*, List<Item>*, st_table*, Procedure*) + 260
0x81d2d3a JOIN::exec() + 4426
0x81d37f5 _Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select + 309
0x81cf048 handle_select(THD*, st_lex*, select_result*, unsigned long) + 260
0x81a1e5a mysql_execute_command(THD*) + 678
0x81a86d9 mysql_parse(THD*, char const*, unsigned int, char const**) + 241
0x81a07c2 dispatch_command(enum_server_command, THD*, char*, unsigned int) + 1198
0x81abb40 do_command(THD*) + 144
0x819f9fb handle_one_connection + 647
0x40045aa7 _end + 933509815
0x40176c2e _end + 934759486

Verified just as described with a bigger private test case uploaded.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/40726

ChangeSet@1.2595, 2008-01-09 00:40:41+03:00, evgen@moonbone.local +3 -0
  Bug#33675: Usage of an uninitialized memory by filesort in a subquery caused
  server crash.
  
  The filesort implementation has an optimization for subquery execution which
  consists of reusing previously allocated buffers. In particular the call to
  the read_buffpek_from_file function might be skipped when a big enough buffer
  for buffer descriptors (buffpeks) is already allocated. Beside allocating
  memory for buffpeks this function fills allocated buffer with data read from
  disk. Skipping it might led to using an arbitrary memory as fields' data and
  finally to a crash.
  
  Now the read_buffpek_from_file function is always called. It allocates
  new buffer only when necessary, but always fill it with correct data.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/40854

ChangeSet@1.2597, 2008-01-10 18:54:34+03:00, evgen@moonbone.local +1 -0
  filesort.cc:
    Bug#33675: Usage of an uninitialized memory by filesort in a subquery caused
    server crash.
    Free smaller buffer before allocating bigger one.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/40857

ChangeSet@1.2779, 2008-01-10 19:26:41+03:00, kaa@kaamos.(none) +3 -0
  Post-merge fix for the patch for bug #33675 after the merge to 6.0.

Pushed into 6.0.5-alpha

Pushed into 5.1.23-rc

Pushed into 5.0.56

Noted in 5.0.56, 5.1.23, 6.0.5 changelogs.

Use of uninitialized memory for filesort in a subquery caused a
server crash.