MySQL Bugs: #42991: invalid memory access and/or crash when using index condition pushdown + innodb

Bug #42991	invalid memory access and/or crash when using index condition pushdown + innodb
Submitted:	18 Feb 20:59	Modified:	19 Feb 8:25
Reporter:	Shane Bester
Status:	Verified
Category:	Server: Optimizer	Severity:	S1 (Critical)
Version:	6.0.9,6.0.12,6.0-bzr	OS:	Any
Assigned to:		Target Version:
Tags:	index_condition_pushdown, optimizer_switch
Triage:	Triaged: D1 (Critical)

Description:
I had an intermittently crashing query, and intermittent valgrind errors at this
callstack:  

mysqld.exe!my_strnncollsp_simple()[ctype-simple.c:166]                     
mysqld.exe!Field_blob::cmp()[field.cc:7729]                                
mysqld.exe!Field_blob::key_cmp()[field.cc:7842]                            
mysqld.exe!key_cmp()[key.cc:478]                                           
mysqld.exe!handler::compare_key2()[handler.cc:5070]                        
mysqld.exe!index_cond_func_innodb()[ha_innodb.cc:8411]                     
mysqld.exe!row_search_for_mysql()[row0sel.c:4193]                          
mysqld.exe!sortcmp()[sql_string.cc:669]                                    
mysqld.exe!srv_conc_enter_innodb()[srv0srv.c:989]                          
mysqld.exe!ha_innobase::index_next()[ha_innodb.cc:4481]                    
mysqld.exe!handler::read_range_next()[handler.cc:5027]                     
mysqld.exe!handler::multi_range_read_next()[handler.cc:4288]               
mysqld.exe!DsMrr_impl::dsmrr_next()[handler.cc:4540]                       
mysqld.exe!ha_innobase::multi_range_read_next()[ha_innodb.cc:8374]         
mysqld.exe!QUICK_RANGE_SELECT::get_next()[opt_range.cc:8539]               
mysqld.exe!rr_quick()[records.cc:322]                                      
mysqld.exe!sub_select()[sql_select.cc:14083]                               
mysqld.exe!do_select()[sql_select.cc:13762]                                
mysqld.exe!JOIN::exec()[sql_select.cc:2439]                                
mysqld.exe!mysql_select()[sql_select.cc:3043]                              
mysqld.exe!handle_select()[sql_select.cc:298]                              
mysqld.exe!execute_sqlcom_select()[sql_parse.cc:4747]                      
mysqld.exe!mysql_execute_command()[sql_parse.cc:2007]                      
mysqld.exe!mysql_parse()[sql_parse.cc:5735]                                
mysqld.exe!dispatch_command()[sql_parse.cc:1009]                           
mysqld.exe!do_command()[sql_parse.cc:690]                                  
mysqld.exe!handle_one_connection()[sql_connect.cc:1145]                    
mysqld.exe!pthread_start()[my_winthread.c:61]                              
mysqld.exe!_callthreadstartex()[threadex.c:348]                            
mysqld.exe!_threadstartex()[threadex.c:326]                                
kernel32.dll!FlsSetValue()                                                 

How to repeat:
run mysqld under valgrind.
import the attached file.
might be tricky to repeat, to try it a few times, using debug and release server...

import a few times against mysqld under valgrind, wait for errors...

Attachment: bug42991.sql (text/x-sql), 63.73 KiB.

full debug infos, valgrind outputs, etc.

Attachment: bug42991_debug_valgrind_infos.txt (text/plain), 10.37 KiB.

Thank you for the report.

Valgrind warnings verified as described:

==14704== Invalid read of size 1
==14704==    at 0x8834A1C: my_strnncollsp_simple (ctype-simple.c:167)
==14704==    by 0x82944C8: Field_blob::cmp(unsigned char const*, unsigned, unsigned char
const*, unsigned) (field.cc:7734)
==14704==    by 0x8294999: Field_blob::key_cmp(unsigned char const*, unsigned)
(field.cc:7847)
==14704==    by 0x82A1A72: key_cmp(st_key_part_info*, unsigned char const*, unsigned)
(key.cc:477)
==14704==    by 0x840B112: handler::compare_key2(st_key_range*) (handler.cc:5127)
==14704==    by 0x868C958: index_cond_func_innodb (ha_innodb.cc:8543)
==14704==    by 0x86F3EFE: row_search_for_mysql (row0sel.c:4143)
==14704==    by 0x8686A5F: ha_innobase::general_fetch(unsigned char*, unsigned, unsigned)
(ha_innodb.cc:4656)
==14704==    by 0x8686B55: ha_innobase::index_next(unsigned char*) (ha_innodb.cc:4692)
==14704==    by 0x840AE79: handler::read_range_next() (handler.cc:5083)
==14704==    by 0x868CA32: ha_innobase::read_range_next() (ha_innodb.cc:8578)
==14704==    by 0x8409558: handler::multi_range_read_next(char**) (handler.cc:4304)
==14704==    by 0x840A084: DsMrr_impl::dsmrr_next(char**) (handler.cc:4599)
==14704==    by 0x868C853: ha_innobase::multi_range_read_next(char**)
(ha_innodb.cc:8505)
==14704==    by 0x83F12F0: QUICK_RANGE_SELECT::get_next() (opt_range.cc:8558)
==14704==    by 0x83FD590: rr_quick(READ_RECORD*) (records.cc:322)
==14704==  Address 0x28438cd0 is 96 bytes inside a block of size 16,464 free'd
==14704==    at 0x400542D: free (vg_replace_malloc.c:323)
==14704==    by 0x8710057: ut_free (ut0mem.c:252)
==14704==    by 0x86CFC9D: mem_area_free (mem0pool.c:472)
==14704==    by 0x86CF069: mem_heap_block_free (mem0mem.c:524)
==14704==    by 0x86CE29C: mem_heap_free_func (mem0mem.ic:487)
==14704==    by 0x86F1C12: row_sel_store_mysql_rec (row0sel.c:2606)
==14704==    by 0x86F3EE7: row_search_for_mysql (row0sel.c:4141)
==14704==    by 0x8686A5F: ha_innobase::general_fetch(unsigned char*, unsigned, unsigned)
(ha_innodb.cc:4656)
==14704==    by 0x8686B55: ha_innobase::index_next(unsigned char*) (ha_innodb.cc:4692)
==14704==    by 0x840AE79: handler::read_range_next() (handler.cc:5083)
==14704==    by 0x868CA32: ha_innobase::read_range_next() (ha_innodb.cc:8578)
==14704==    by 0x8409558: handler::multi_range_read_next(char**) (handler.cc:4304)
==14704==    by 0x840A084: DsMrr_impl::dsmrr_next(char**) (handler.cc:4599)
==14704==    by 0x868C853: ha_innobase::multi_range_read_next(char**)
(ha_innodb.cc:8505)
==14704==    by 0x83F12F0: QUICK_RANGE_SELECT::get_next() (opt_range.cc:8558)
==14704==    by 0x83FD590: rr_quick(READ_RECORD*) (records.cc:322)

Got another crash with engine_condition_pushdown=1:

Thread 11:
Invalid write of size 4
at:  mem_pool_fill_free_list (mem0pool.c:302)
by: mem_pool_fill_free_list (mem0pool.c:286)
by: mem_area_alloc (mem0pool.c:346)
by: mem_heap_create_block (mem0mem.c:370)
by: mem_heap_add_block (mem0mem.c:465)
by: lock_rec_create (mem0mem.ic:155)
by: lock_rec_lock (lock0lock.c:1972)
by: lock_clust_rec_read_check_and_lock (lock0lock.c:5159)
by: row_search_for_mysql (row0sel.c:851)
by: ha_innobase::general_fetch (ha_innodb.cc:4671)
by: ha_innobase::rnd_next(unsigned char*) (ha_innodb.cc:4860)
by: rr_sequential(READ_RECORD*) (records.cc:390)
Address 0x10 is not stack'd, malloc'd or (recently) free'd

goes away / comes back when disabling/enabling ICP in InnoDB

If I add EXPLAIN to the testcase, I get this:

+explain select * from `table5` where  (col2 <= '6566-06-15' AND col24 <> 'd')  group by
`col83` order by `col83` desc  ;
+id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
+1	SIMPLE	table5	range	PRIMARY,idx1,idx3,idx4	idx3	89	NULL	1	Using index condition; Using
where; Using temporary; Using filesort

it is using idx3 which is defined as 

  KEY `idx3` (`col2`(86))

push_index_cond() and co correctly detect that "col2 <= '6566-06-15" part of the
condition cannot be checked by using the index.
The "col24 <> 'd'" part does get pushed (col24 is a clustered primary key, so it is part
of the index).  Since index condition is present, it will try to decode the entire index
tuple. I didn't investigate what exactly goes wrong when that is done, but it can be
wrong on various levels as we don't ever try decoding blob columns from index images of
their prefixes.

Suggested solution right now is to disable index condition for indexes that have
partially-covered columns (check key_part->flag & HA_PART_KEY_SEG for all key parts?)