Bug #108324 Easy to OOM mysqld with innodb_dedicated_server=ON and Innodb cluster
Submitted: 29 Aug 2022 16:03 Modified: 8 Sep 2022 9:52
Reporter: Jay Janssen Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Server Severity:S3 (Non-critical)
Version:8.0.30 OS:Red Hat
Assigned to: MySQL Verification Team CPU Architecture:x86

[29 Aug 2022 16:03] Jay Janssen
Description:
Running Innodb cluster nodes with innodb-dedicated-server=on, I can easily OOM cluster members loading sysbench data.

This is a server reporting 29G of memory via `free -h`.  The innodb buffer pool is set (automatically) to 24696061952.  I have seen this on other servers with differing amounts of memory, so I think it's a consistent problem.  

I can OOM servers in the cluster simply running this sysbench prepare:

sysbench --table-size=10000000 --tables=20 --threads=4 --time=0 --report-interval=1 --mysql-host=127.0.0.1 --mysql-port=6446 --mysql-user=sbtest --mysql-password=PASSWORD --db-driver=mysql /usr/share/sysbench/oltp_read_write.lua prepare

How to repeat:
1. Setup an Innodb cluster using 'innodb-dedicated-server=on'
2. Run the above sysbench prepare

Suggested fix:
adjust innodb-dedicated-server to take Group replication/Innodb cluster into consideration somehow.
[6 Sep 2022 4:32] MySQL Verification Team
Hi,

I am not sure how "easy" you can OOM the server. I tried for a while and could not do it on a much smaller server?

Can you share full config?

thanks
[7 Sep 2022 13:11] Jay Janssen
The configuration is not complex at all.  

[mysqld]
datadir=/data/mysql
socket=/data/mysql/mysql.sock

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

!includedir /etc/my.cnf.d

skip-name-resolve

# Innodb
innodb-dedicated-server=on

# Replication
binlog_expire_logs_seconds=604800

# Cluster / Group replication
loose_group_replication_paxos_single_leader=ON

This is using Amazon Linux 2 with no tuning.  

I've added innodb-buffer-pool-size=21565865984, but still had OOM issues.  

When I set sysctl vm.swappiness=0, and oom_score_adj=-1000 things seem more stable.  I'm wondering if I should try removing my innodb-buffer-pool-size override.
[7 Sep 2022 14:39] Jay Janssen
Assuming your environment is properly setting swappiness, I'd be fine closing this bug.
[8 Sep 2022 9:52] MySQL Verification Team
Hi,

> When I set sysctl vm.swappiness=0, and oom_score_adj=-1000 things seem 
> more stable. 
> I'm wondering if I should try removing my innodb-buffer-pool-size override.

IMHO you should always have innodb_buffer_pool_size in the config and have it not higher than 70% of the total RAM.

> Assuming your environment is properly setting swappiness, 

I personally run 0 or 1. Linux kernel changes behavior for swappiness through time and even today different distros can have different behavior so for me the safe value on MySQL Server machines is 0 or 1 (I prefer 0). Especially on EC2 I had some nasty issues irrelevant from MySQL with swappiness so for EC2 I personally force 0. On old kernel I used 10 but these days 0 is mine preferred value on servers where MySQL is running.

> I'd be fine closing this bug.

Ok. Closed, as this is both configuration issue and possible issue with Amazon linux.

kind regards