Bug #118276 Resource Groups can't be usable inside with CPU limited Containers (cgroups/cpuset)
Submitted: 27 May 5:20 Modified: 27 May 8:02
Reporter: Soju Yamashita Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: DDL Severity:S4 (Feature request)
Version:8.4.4 OS:Linux
Assigned to: CPU Architecture:x86

[27 May 5:20] Soju Yamashita
Description:
We run MySQL in a Kubernetes Pod whose CPU quota is limited and pinned to a subset of the host CPUs (for example 3,4,6,7) by means of the cpuset cgroup.
Pinning a pod to specific CPUs is a common performance recommendation when CPU resources are capped.
(see https://www.cockroachlabs.com/docs/stable/kubernetes-performance#:~:text=Warning%3A,the%20.... )

In such a environment, MySQL detects for CPUs numbered 0-3 instead of actual host IDs(3,4,6,7). 
Creating a resource group that refers to the real CPU IDs and binding a thread to it fails. 
The behaviour appears identical to the previously reported bug #99274 (https://bugs.mysql.com/bug.php?id=99274).
Because the limitation is applied through cgroups, the same problem can be reproduced in any environment that starts MySQL with cpuset configuration (e.g. a plain Docker container).
I think it is also expected to occur when using mysql/mysql-operator.

How to repeat:
Host machine have 8 CPUs, (0-7)
build a Docker Container and give CPU 3,4,6,7 to this container.

1. Deploy MySQL container with CPU limited.
The docker-compose configuration files are as follows.
```
services:
  database:
    container_name: db
    image: container-registry.oracle.com/mysql/community-server:8.4.4
    cap_add:
      - SYS_NICE
    cpuset: 3,4,6,7
    environment:
      - MYSQL_ALLOW_EMPTY_PASSWORD=true
```

2. We can't create/alter a Resource Group with real CPU ID such as 3,4 because these are out of CPU range expected by mysql

```
mysql> CREATE RESOURCE GROUP low type=user;
mysql> ALTER RESOURCE GROUP low VCPU=3,4;
ERROR 3652 (HY000): Invalid cpu id 4
```

3. we can create Resource Group with CPU ID below 4(because mysql detect 4 cpus), such as CPU 0, but we can't set this resource group to a thread. Because Host OS didn't give this CPU ID to Container:

```
mysql> ALTER RESOURCE GROUP low VCPU=0;
Query OK, 0 rows affected (0.00 sec)
mysql> set resource group low;
ERROR 3661 (HY000): Unable to bind resource group low with thread id (44).(Failed to apply thread resource controls).
```

4. If I create a resource group with cpu ID 3, I can bind this rg to thread because this CPU ID is given by Host OS and in CPU range expected.
```
mysql> ALTER RESOURCE GROUP low VCPU=3;
Query OK, 0 rows affected (0.00 sec)
mysql> set resource group low;
Query OK, 0 rows affected (0.01 sec)

Suggested fix:
1. validate_vcpu_range_vector: do not check valid cpu id by num_cpus, but check if the CPU ID is owned by mysql-server
2. resource_group_mgr: Store not only the number of CPUs but also the explicit list of CPU IDs available to the MySQL server, and use this list during validation and binding.
[27 May 8:02] MySQL Verification Team
Hello Soju Yamashita,

Thank you for the feature request!

regards,
Umesh