MySQL Bugs: #106547: Access to btr_search_sys->hash

Bug #106547	Access to btr_search_sys->hash_tables is not safe
Submitted:	23 Feb 2022 9:17	Modified:	24 Feb 2022 13:54
Reporter:	Baolin Huang	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S3 (Non-critical)
Version:	8.0	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
1. In function btr_search_info_update_slow, access hash_tables under no ahi latch.

2. In function btr_search_sys_resize, the hash_tables is freed and recreated.

There is a probability a null pointer can be accessed.

How to repeat:
 I will provide a test case later, if necessary.

Hi Mr. Huang,

Thank you for your bug report.

However, we do not see a problem that you are reporting.

Hence, we are eagerly awaiting your fully repeatable test case !!!!

Hello Baolin Huang, could you also provide the call path through which `btr_search_info_update_slow` can access `hash_tables`?

AFAICT neither `btr_search_info_update_hash` nor `btr_search_update_block_hash_info` access it.

Here is a way to reproduce.
1. Need to add some code, i was based on 8.0.25, the diff is on below.

2. start mysql server with 
--innodb_adaptive_hash_index_parts=1 
--innodb_adaptive_hash_index=on
--innodb_buffer_pool_instances=1

3. Run sysbench  prepare 
sysbench  --mysql-user=sbtest --mysql-password=sbtest_123 --mysql-db=test oltp_write_only --tables=20 --table_size=10000 --threads=32 --time=3600 prepare

4. set global debug='+d,core_on_check_free';

5. sysbench  --mysql-user=sbtest --mysql-password=sbtest_123 --mysql-db=test oltp_write_only --tables=20 --table_size=10000 --threads=32 --time=3600 run

6. set global innodb_buffer_pool_size=25165824*10;

7. wait the server crash

```
diff --git a/storage/innobase/btr/btr0sea.cc b/storage/innobase/btr/btr0sea.cc
index 3be5561d4e4..ef232a03a5c 100644
--- a/storage/innobase/btr/btr0sea.cc
+++ b/storage/innobase/btr/btr0sea.cc
@@ -159,8 +159,10 @@ static void btr_search_check_free_space_in_heap(dict_index_t *index) {
 
   ut_ad(!rw_lock_own(btr_get_search_latch(index), RW_LOCK_S));
   ut_ad(!rw_lock_own(btr_get_search_latch(index), RW_LOCK_X));
+  DBUG_EXECUTE_IF("core_on_check_free", std::this_thread::sleep_for(std::chrono::microseconds(10000000)););
 
   table = btr_get_search_table(index);
+  ut_a(table);
 
   heap = table->heap;
 
@@ -224,6 +226,7 @@ void btr_search_sys_create(ulint hash_size) {
 @param[in]     hash_size       hash index hash table size */
 void btr_search_sys_resize(ulint hash_size) {
   /* Step-1: Lock all search latches in exclusive mode. */
+
   btr_search_x_lock_all();
 
   if (btr_search_enabled) {
@@ -240,6 +243,8 @@ void btr_search_sys_resize(ulint hash_size) {
     mem_heap_free(btr_search_sys->hash_tables[i]->heap);
     hash_table_free(btr_search_sys->hash_tables[i]);
 
+    btr_search_sys->hash_tables[i] = nullptr;
+    DBUG_EXECUTE_IF("core_on_check_free", std::this_thread::sleep_for(std::chrono::microseconds(20000000)););
     btr_search_sys->hash_tables[i] =
         ib_create((hash_size / btr_ahi_parts), LATCH_ID_HASH_TABLE_MUTEX, 0,
                   MEM_HEAP_FOR_BTR_SEARCH);
```

Crash stack is 
#0  0x00007f3a72542387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f3a72543a78 in __GI_abort () at abort.c:90
#2  0x00000000053e77cd in ut_dbg_assertion_failed (expr=0x7982ba3 "table",     file=0x79844a8 "storage/innobase/btr/btr0sea.cc", line=165)    at storage/innobase/ut/ut0dbg.cc:99
#3  0x0000000005479169 in btr_search_check_free_space_in_heap (index=0x7f39180fad70)    at storage/innobase/btr/btr0sea.cc:165
#4  0x000000000547aa58 in btr_search_info_update_slow (info=0x7f39180fb228, cursor=0x7f3a387ad6c0)    at storage/innobase/btr/btr0sea.cc:679
#5  0x000000000545df52 in btr_search_info_update (index=0x7f39180fad70, cursor=0x7f3a387ad6c0)    at storage/innobase/include/btr0sea.ic:85
#6  0x0000000005464005 in btr_cur_search_to_nth_level (index=0x7f39180fad70, level=0, tuple=0x7f39342e6ab0,     mode=PAGE_CUR_LE, latch_mode=2, cursor=0x7f3a387ad6c0, has_search_latch=0,     file=0x78e8a10 "storage/innobase/row/row0ins.cc", line=2408,     mtr=0x7f3a387ad1a0) at storage/innobase/btr/btr0cur.cc:1651
#7  0x000000000515bc3d in btr_pcur_t::open (this=0x7f3a387ad6c0, index=0x7f39180fad70, level=0,     tuple=0x7f39342e6ab0, mode=PAGE_CUR_LE, latch_mode=2, mtr=0x7f3a387ad1a0,     file=0x78e8a10 "storage/innobase/row/row0ins.cc", line=2408)    at storage/innobase/include/btr0pcur.h:646
#8  0x0000000005296e7b in row_ins_clust_index_entry_low (flags=0, mode=2, index=0x7f39180fad70, n_uniq=1,     entry=0x7f39342e6ab0, thr=0x7f39342e4fa0, dup_chk_only=false)    at storage/innobase/row/row0ins.cc:2408
#9  0x000000000529912e in row_ins_clust_index_entry (index=0x7f39180fad70, entry=0x7f39342e6ab0,     thr=0x7f39342e4fa0, dup_chk_only=false) at storage/innobase/row/row0ins.cc:3097
#10 0x000000000529987d in row_ins_index_entry (index=0x7f39180fad70, entry=0x7f39342e6ab0,     multi_val_pos=@0x7f3940036bd8: 0, thr=0x7f39342e4fa0)    at storage/innobase/row/row0ins.cc:3289
#11 0x0000000005299e53 in row_ins_index_entry_step (node=0x7f3940036b18, thr=0x7f39342e4fa0)    at storage/innobase/row/row0ins.cc:3425
#12 0x000000000529a1d2 in row_ins (node=0x7f3940036b18, thr=0x7f39342e4fa0)
    at storage/innobase/row/row0ins.cc:3545

Hello Baolin!
Thank you for more detailed report. Indeed there is an issue, and it is with btr_get_search_table() - it should be accessed only when having an appropriate ahi part latched (either S- or X-latched). The same problem is visible when calling btr_get_search_table() without latches in the following methods:
btr_search_check_free_space_in_heap()
btr_search_update_hash_on_delete()
btr_search_update_hash_on_insert()

This problem was found by the InnoDB team last month and is being fixed as part of work on https://bugs.mysql.com/bug.php?id=100512 . This bug will be closed together with the mentioned one.

Thank you for reporting this issue!

Hi Mr. Huang,

Thank you for providing us with all feedback info that we have asked for.

This has enabled us to establish that your report is a duplicate of the following one:

https://bugs.mysql.com/bug.php?id=100512

We do not know when will bug be fixed, but you can subscribe to it and then you will find out which release will have this problem fixed.

Thanks again .......