Bug #119585 In extreme cases, sync_array_get_and_reserve_cell fails to obtain a cell even have space

Description:
In extreme cases, there may be free cells in the sync_wait_array, but the function sync_array_get_and_reserve_cell fails to acquire one, leading to an abnormal crash.

The function sync_array_get_and_reserve_cell calls sync_array_get to obtain an array from the sync_wait_array.

The following code attempts to acquire elements from the sync_wait_array through a loop. However, it cannot guarantee that all elements will be accessed within the loop—some may be accessed multiple times, and in extreme cases, the same element may be retrieved every time:

for (ulint i = 0; i < sync_array_size && *cell == nullptr; ++i) {
    /* Although the sync_array is get in a random way currently,                                      
    we still try at most sync_array_size times, in case any                                           
    of the sync_array we get is full */
    sync_arr = sync_array_get();
    *cell = sync_array_reserve_cell(sync_arr, object, type, location);
}

If the cells of this particular element are exhausted, it will ultimately lead to a crash upon exiting the loop.

How to repeat:
Modify the following function so that default_indexer_t<>::get_rnd_index() always returns a fixed value to simulate the extreme cases.

static inline sync_array_t *sync_array_get() {
  if (sync_array_size <= 1) {
    return (sync_wait_array[0]);
  }

  return (
      sync_wait_array[default_indexer_t<>::get_rnd_index() % sync_array_size]);
}

Suggested fix:
use a random positon and visit all the sync_wait_array element in a loop from this position

Thank you for the bug report. The functions you refer to seem to be in the InnoDB code, so I'm changing the bug category to InnoDB.

Just a bit more detailed explanation:

sync_array_get uses get_rnd_index, which is based on my_timer_cycles, which in turn is based on RDTSC instruction; In older CPU microarchitectures RDTSC would return values that increment by 1 (per-core/per-cycle), but in more modern CPUs it will return (for example) multiples of 32.  This is a problem because when you use modulo on these values, they will split into a limited number of "buckets", which in the case of sync_array_get results in MUCH higher probability of hitting https://bugs.mysql.com/bug.php?id=119567

As I was doing a bit of research I found this interesting reference table for rdtsc increments in different AMD microarchitectures (quoting from https://inria.hal.science/hal-02866777v1/document):
CPU                 μ-arch.                     Increment
=================== =========================== =========
AMD Athlon 64       X2 3800+ K8                         1
AMD Turion II       Neo N40L K10                        1
AMD Phenom II       X6 1055T K10                        1
AMD E-450           Bobcat                              1
AMD Athlon          5350 Jaguar                         1
AMD FX-4100         Bulldozer                           1
AMD FX-8350         Piledriver                          1
AMD A10-7870K       Steamroller                         1
AMD Ryzen           Threadripper 1920X Zen             35
AMD Ryzen           Threadripper 1950X Zen             34
AMD Ryzen           Threadripper 1700X Zen             34
AMD Ryzen           Threadripper 2970WX Zen+           30
AMD Ryzen 7 3700X   Zen 2                              36
AMD EPYC 7401p      Zen                                20
AMD EPYC 7571       Zen                                22

So you can see that the my_timer_cycles will be tied to the CPU architecture.

To "repeat" (or rather to verify the problem is there):
========================================================

Set innodb_sync_array_size=1024 and run some simple by contentious load; For example:

~$ sysbench $auth --db-driver=mysql  --time=0 --rand-type=uniform  --table-size=2 --tables=1 --threads=512 --report-interval=5 --create_secondary=on /usr/share/sysbench/oltp_read_write.lua run;

Then observe the sync arrays via first parameter to sync_array_reserve_cell:
~$  sudo bpftrace -e 'uprobe:/home/kolita/opt/percona/8.4.7/bin/mysqld:_Z23sync_array_reserve_cellP12sync_array_tPvmN2ut8LocationE { printf("arr: %u (%u)\n",arg0,arg0%32); }'
Attaching 1 probe...
arr: 3875288064 (0)
arr: 3869525952 (0)
arr: 3867788288 (0)
arr: 3871258304 (0)
arr: 3868232576 (0)
arr: 3871685504 (0)
arr: 3868664672 (0)
arr: 3874429664 (0)
arr: 3873562496 (0)
arr: 3869959648 (0)
arr: 3867789248 (0)
arr: 3871254784 (0)
arr: 3871255424 (0)
arr: 3869962368 (0)
arr: 3867363424 (0)
arr: 3867364064 (0)
arr: 3873561856 (0)
arr: 3874431744 (0)

Notice how modulo 32 always returns zero, which means only 32 of the 1024 sync array entries will be used.

Fix (?):
=======
Use a different source for get_rnd_index (https://github.com/mysql/mysql-server/blob/845d525d49c8027a4d0cdcc43372c96ba295c857/storag...) so that it doesn't depend on architecture implementations.