Bug #110084 MySQL Operator: mysql-operator does not update my.cnf
Submitted: 15 Feb 2023 16:10 Modified: 21 Feb 2023 16:42
Reporter: Vy Nguyen Tan Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Operator Severity:S3 (Non-critical)
Version:2.0.8 - 8.0.32 OS:Any
Assigned to: CPU Architecture:Any

[15 Feb 2023 16:10] Vy Nguyen Tan
Description:
mysql-operator does not update my.cnf after updating new serverConfig.mycnf value

How to repeat:
- Step 1: Create a new innodb cluster via mysql-operator/mysql-innodbcluster chart.

helmfile.yaml:
---------------
repositories:
  - name: mysql-operator
    url: "https://mysql.github.io/mysql-operator"
releases:
  - name: mysql-innodbcluster-p1
    namespace: infra-databases
    chart: mysql-operator/mysql-innodbcluster
    version: 2.0.8
    installed: true
    values:
      - values/mysql-innodbcluster-p1-config.yaml 

values/mysql-innodbcluster-p1-config.yaml:
---------
...
serverConfig:
  mycnf: |
    [mysqld]
    core_file
    local_infile=off
    max_connections=1000
    max_allowed_packet=2048M
...

get max_connections:
---------

$ kubectl -n infra-databases exec -it mysql-innodbcluster-p1-0 mysql --container mysql -- cat /etc/my.cnf.d/99-extra.cnf
# Additional user configurations taken from spec.mycnf in InnoDBCluster.
# Do not edit directly.
[mysqld]
core_file
local_infile=off
max_connections=1000
max_allowed_packet=2048M

$ kubectl -n infra-databases exec -it mysql-innodbcluster-p1-1 mysql --container mysql -- cat /etc/my.cnf.d/99-extra.cnf
# Additional user configurations taken from spec.mycnf in InnoDBCluster.
# Do not edit directly.
[mysqld]
core_file
local_infile=off
max_connections=1000
max_allowed_packet=2048M

$ kubectl -n infra-databases exec -it mysql-innodbcluster-p1-0 mysql --container mysql -- mysql -u root -pxxxxx -e 'select @@max_connections'
mysql: [Warning] Using a password on the command line interface can be insecure.
+-------------------+
| @@max_connections |
+-------------------+
|              1000 |
+-------------------+

$ kubectl -n infra-databases exec -it mysql-innodbcluster-p1-1 mysql --container mysql -- mysql -u root -pxxxx -e 'select @@max_connections'
mysql: [Warning] Using a password on the command line interface can be insecure.
+-------------------+
| @@max_connections |
+-------------------+
|              1000 |
+-------------------+

- Step 2: change max_connections

values/mysql-innodbcluster-p1-config.yaml:
---------
...
serverConfig:
  mycnf: |
    [mysqld]
    core_file
    local_infile=off
    max_connections=2000
    max_allowed_packet=2048M
...

appy new config
---------
$ helmfile -f helmfile.yaml apply

- Step 3: get max_connections

$ kubectl -n infra-databases exec -it mysql-innodbcluster-p1-0 mysql --container mysql -- cat /etc/my.cnf.d/99-extra.cnf
# Additional user configurations taken from spec.mycnf in InnoDBCluster.
# Do not edit directly.
[mysqld]
core_file
local_infile=off
max_connections=1000
max_allowed_packet=2048M

$ kubectl -n infra-databases exec -it mysql-innodbcluster-p1-1 mysql --container mysql -- cat /etc/my.cnf.d/99-extra.cnf
# Additional user configurations taken from spec.mycnf in InnoDBCluster.
# Do not edit directly.
[mysqld]
core_file
local_infile=off
max_connections=1000
max_allowed_packet=2048M

$ kubectl -n infra-databases exec -it mysql-innodbcluster-p1-0 mysql --container mysql -- mysql -u root -pxxxxx -e 'select @@max_connections'
mysql: [Warning] Using a password on the command line interface can be insecure.
+-------------------+
| @@max_connections |
+-------------------+
|              1000 |
+-------------------+

$ kubectl -n infra-databases exec -it mysql-innodbcluster-p1-1 mysql --container mysql -- mysql -u root -pxxxxx -e 'select @@max_connections'
mysql: [Warning] Using a password on the command line interface can be insecure.
+-------------------+
| @@max_connections |
+-------------------+
|              1000 |
+-------------------+
[21 Feb 2023 16:42] MySQL Verification Team
Hi,

MySQL should not edit config file IIRC. We do "permanent config changes" in mysqld-auto.cnf found in datadir location. Check documentation on "persisted-globals": 

https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_persisted_glob...

https://dev.mysql.com/doc/refman/8.0/en/persisted-system-variables.html

I'm not sure what MySQL operator does and what's the proper way to handle this. I'll verify the report for now and we'll see what operator team will decide.

Thank you for the report.
[13 Apr 2023 2:32] Avash Kumar
hi

I am from IBM India. I work in a commercial project where we use mysql operator.
Even we faced the same issue. 

We did a workaround where manually 
1. we updated the mycnf section in Cluster.yaml file
2. manually updated the same values in the configmap 'initconf' which was created at time of cluster creation
3. restarted the mysql pods one by one (with mysql master restarted last) for the updated mysql values to take into effect
[14 Sep 2023 14:55] Scott Anderson
We are experiencing this exact same issue on the 8.1.0.  Any updates on this issue.

Best regards
 
Scott Anderson
Senior Platform Architect
[21 Sep 2023 11:15] Lukas Lösche
Can confirm this is still an issue in 8.1. Any changes to spec.mycnf are not being propagated to the cluster.
[7 Mar 11:55] Carlos Abrantes
Still happening on MySQL Operator 8.3.0-2.1.2 from  (2024-01-16).

Since change MySQL config is a pretty strong use case, shouldn't this be fixed already? any ETA expected? I would expect such basic feature available for a production ready Operator or the usage of MySQL operator should not be consider for production?

Thanks,