Description:
Upgrade InnoDBcluster is not updating the cluster pods from cluster.yaml
Ex: Initially we have deployed the cluster with basic configuration.. and later we tried to update cluster with modified configuration .. but cluster pods are not updated with latest configuration.
Basic configuration:
apiVersion: mysql.oracle.com/v2
kind: InnoDBCluster
metadata:
name: mycluster
spec:
secretName: mypwds
instances: 3
router:
instances: 1
tlsUseSelfSigned: true
How to repeat:
Modified Innodb cluster
configuration:
apiVersion: mysql.oracle.com/v2
kind: InnoDBCluster
metadata:
name: mycluster
spec:
secretName: mypwds
instances: 3
router:
instances: 1
tlsUseSelfSigned: true
datadirVolumeClaimTemplate:
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
selector:
matchLabels:
type: local
podSpec:
containers:
- name: mysql
resources:
requests:
memory: "1048Mi"
cpu: "1000m"
limits:
memory: "1192Mi"
cpu: "1200m"
kubectl apply -f sample-cluster-podspec-resources.yaml -n mycluster
kubectl describe pod mycluster-0 -n mycluster
Name: mycluster-0
Namespace: mycluster
Priority: 0
Service Account: mycluster-sidecar-sa
Node: minikube/192.168.49.2
Start Time: Thu, 04 Jul 2024 15:42:25 +0530
Labels: app.kubernetes.io/component=database
app.kubernetes.io/created-by=mysql-operator
app.kubernetes.io/instance=mysql-innodbcluster-mycluster-mysql-server
app.kubernetes.io/managed-by=mysql-operator
app.kubernetes.io/name=mysql-innodbcluster-mysql-server
apps.kubernetes.io/pod-index=0
component=mysqld
controller-revision-hash=mycluster-767954bbfd
mysql.oracle.com/cluster=mycluster
mysql.oracle.com/cluster-role=PRIMARY
mysql.oracle.com/instance-type=group-member
statefulset.kubernetes.io/pod-name=mycluster-0
tier=mysql
Annotations: kopf.zalando.org/last-handled-configuration:
{"spec":{"volumes":[{"name":"datadir","persistentVolumeClaim":{"claimName":"datadir-mycluster-0"}},{"name":"mycnfdata","emptyDir":{}},{"na...
mysql.oracle.com/fqdn-template: {service}.{namespace}.svc.{domain}
mysql.oracle.com/membership-info:
{"memberId": "fd265094-39ed-11ef-a354-a61ef5709950", "lastTransitionTime": "2024-07-04T10:25:41Z", "lastProbeTime": "2024-07-04T10:25:41Z"...
Status: Running
IP: 10.244.0.4
IPs:
IP: 10.244.0.4
Controlled By: StatefulSet/mycluster
Init Containers:
fixdatadir:
Container ID: docker://5624e37fdb14d70cb75a350b00de31de5b7345933ebc29a7ae18e0a48f0cad9e
Image: container-registry.oracle.com/mysql/community-operator:9.0.0-2.2.0
Image ID: docker-pullable://container-registry.oracle.com/mysql/community-operator@sha256:1daaad9a004e1e94e4fed1c5b56074e295c507cc7ff51cf38e89e8edef0263f6
Port: <none>
Host Port: <none>
Command:
bash
-c
chown 27:27 /var/lib/mysql && chmod 0700 /var/lib/mysql
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 04 Jul 2024 15:42:27 +0530
Finished: Thu, 04 Jul 2024 15:42:27 +0530
Ready: True
Restart Count: 0
Environment:
MYSQL_OPERATOR_K8S_CLUSTER_DOMAIN: cluster.local
MYSQLSH_CREDENTIAL_STORE_SAVE_PASSWORDS: never
Mounts:
/var/lib/mysql from datadir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro)
initconf:
Container ID: docker://c92d8661be8fea881f052498f5d1947538b801f2a61cdc5dcac75527ec88c0e9
Image: container-registry.oracle.com/mysql/community-operator:9.0.0-2.2.0
Image ID: docker-pullable://container-registry.oracle.com/mysql/community-operator@sha256:1daaad9a004e1e94e4fed1c5b56074e295c507cc7ff51cf38e89e8edef0263f6
Port: <none>
Host Port: <none>
Command:
mysqlsh
--log-level=@INFO
--pym
mysqloperator
init
--pod-name
$(POD_NAME)
--pod-namespace
$(POD_NAMESPACE)
--datadir
/var/lib/mysql
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 04 Jul 2024 15:42:30 +0530
Finished: Thu, 04 Jul 2024 15:42:40 +0530
Ready: True
Restart Count: 0
Environment:
POD_NAME: mycluster-0 (v1:metadata.name)
POD_NAMESPACE: mycluster (v1:metadata.namespace)
MYSQL_OPERATOR_K8S_CLUSTER_DOMAIN: cluster.local
MYSQLSH_USER_CONFIG_HOME: /tmp
MYSQLSH_CREDENTIAL_STORE_SAVE_PASSWORDS: never
Mounts:
/mnt/initconf from initconfdir (ro)
/mnt/mycnfdata from mycnfdata (rw)
/tmp from initconf-tmp (rw)
/var/lib/mysql from datadir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro)
initmysql:
Container ID: docker://a2fced6c6c444a9951ba7b93c7b1e138b9c6efbdc4ab8a39192961aa4d659893
Image: container-registry.oracle.com/mysql/community-server:9.0.0
Image ID: docker-pullable://container-registry.oracle.com/mysql/community-server@sha256:0d6dfa57e104bddf057d5a3ebee8279dbc737cfc32208bbff29c3977a7b73ddb
Port: <none>
Host Port: <none>
Args:
mysqld
--user=mysql
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 04 Jul 2024 15:42:58 +0530
Finished: Thu, 04 Jul 2024 15:43:22 +0530
Ready: True
Restart Count: 0
Environment:
MYSQL_INITIALIZE_ONLY: 1
MYSQL_ROOT_PASSWORD: <set to the key 'rootPassword' in secret 'mypwds'> Optional: false
MYSQLSH_USER_CONFIG_HOME: /tmp
Mounts:
/docker-entrypoint-initdb.d from mycnfdata (rw,path="docker-entrypoint-initdb.d")
/etc/my.cnf from mycnfdata (rw,path="my.cnf")
/etc/my.cnf.d from mycnfdata (rw,path="my.cnf.d")
/tmp from initmysql-tmp (rw)
/var/lib/mysql from datadir (rw)
/var/lib/mysql-files from varlibmysqlfiles (rw)
/var/run/mysqld from rundir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro)
Containers:
sidecar:
Container ID: docker://083922497e4d1f05505e6563e2ac0c416c726195363438b8d9243701856fae2b
Image: container-registry.oracle.com/mysql/community-operator:9.0.0-2.2.0
Image ID: docker-pullable://container-registry.oracle.com/mysql/community-operator@sha256:1daaad9a004e1e94e4fed1c5b56074e295c507cc7ff51cf38e89e8edef0263f6
Port: <none>
Host Port: <none>
Command:
mysqlsh
--pym
mysqloperator
sidecar
--pod-name
$(POD_NAME)
--pod-namespace
$(POD_NAMESPACE)
--datadir
/var/lib/mysql
State: Running
Started: Thu, 04 Jul 2024 15:43:24 +0530
Ready: True
Restart Count: 0
Environment:
POD_NAME: mycluster-0 (v1:metadata.name)
POD_NAMESPACE: mycluster (v1:metadata.namespace)
MYSQL_UNIX_PORT: /var/run/mysqld/mysql.sock
MYSQLSH_USER_CONFIG_HOME: /mysqlsh
MYSQL_OPERATOR_K8S_CLUSTER_DOMAIN: cluster.local
MYSQLSH_CREDENTIAL_STORE_SAVE_PASSWORDS: never
Mounts:
/etc/my.cnf from mycnfdata (rw,path="my.cnf")
/etc/my.cnf.d from mycnfdata (rw,path="my.cnf.d")
/mysqlsh from shellhome (rw)
/tmp from sidecar-tmp (rw)
/var/run/mysqld from rundir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro)
mysql:
Container ID: docker://0eeb2af4b3ff5991fa16ff3deeb1238b2328ba1177c2d803c5a6c32bd8925b9a
Image: container-registry.oracle.com/mysql/community-server:9.0.0
Image ID: docker-pullable://container-registry.oracle.com/mysql/community-server@sha256:0d6dfa57e104bddf057d5a3ebee8279dbc737cfc32208bbff29c3977a7b73ddb
Ports: 3306/TCP, 33060/TCP, 33061/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
mysqld
--user=mysql
State: Running
Started: Thu, 04 Jul 2024 15:43:24 +0530
Ready: True
Restart Count: 0
Liveness: exec [/livenessprobe.sh] delay=15s timeout=1s period=15s #success=1 #failure=10
Readiness: exec [/readinessprobe.sh] delay=10s timeout=1s period=5s #success=1 #failure=10000
Startup: exec [/livenessprobe.sh 8] delay=5s timeout=1s period=3s #success=1 #failure=10000
Environment:
MYSQL_UNIX_PORT: /var/run/mysqld/mysql.sock
MYSQLSH_CREDENTIAL_STORE_SAVE_PASSWORDS: never
Mounts:
/etc/my.cnf from mycnfdata (rw,path="my.cnf")
/etc/my.cnf.d from mycnfdata (rw,path="my.cnf.d")
/livenessprobe.sh from initconfdir (rw,path="livenessprobe.sh")
/readinessprobe.sh from initconfdir (rw,path="readinessprobe.sh")
/tmp from mysql-tmp (rw)
/var/lib/mysql from datadir (rw)
/var/lib/mysql-files from varlibmysqlfiles (rw)
/var/run/mysqld from rundir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro)
Readiness Gates:
Type Status
mysql.oracle.com/configured True
mysql.oracle.com/ready True
Conditions:
Type Status
mysql.oracle.com/ready True
mysql.oracle.com/configured True
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
datadir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: datadir-mycluster-0
ReadOnly: false
mycnfdata:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
rundir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
varlibmysqlfiles:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
initconfdir:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mycluster-initconf
Optional: false
shellhome:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
initconf-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
initmysql-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
mysql-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
sidecar-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-hz6tn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
Description: Upgrade InnoDBcluster is not updating the cluster pods from cluster.yaml Ex: Initially we have deployed the cluster with basic configuration.. and later we tried to update cluster with modified configuration .. but cluster pods are not updated with latest configuration. Basic configuration: apiVersion: mysql.oracle.com/v2 kind: InnoDBCluster metadata: name: mycluster spec: secretName: mypwds instances: 3 router: instances: 1 tlsUseSelfSigned: true How to repeat: Modified Innodb cluster configuration: apiVersion: mysql.oracle.com/v2 kind: InnoDBCluster metadata: name: mycluster spec: secretName: mypwds instances: 3 router: instances: 1 tlsUseSelfSigned: true datadirVolumeClaimTemplate: spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 10Gi selector: matchLabels: type: local podSpec: containers: - name: mysql resources: requests: memory: "1048Mi" cpu: "1000m" limits: memory: "1192Mi" cpu: "1200m" kubectl apply -f sample-cluster-podspec-resources.yaml -n mycluster kubectl describe pod mycluster-0 -n mycluster Name: mycluster-0 Namespace: mycluster Priority: 0 Service Account: mycluster-sidecar-sa Node: minikube/192.168.49.2 Start Time: Thu, 04 Jul 2024 15:42:25 +0530 Labels: app.kubernetes.io/component=database app.kubernetes.io/created-by=mysql-operator app.kubernetes.io/instance=mysql-innodbcluster-mycluster-mysql-server app.kubernetes.io/managed-by=mysql-operator app.kubernetes.io/name=mysql-innodbcluster-mysql-server apps.kubernetes.io/pod-index=0 component=mysqld controller-revision-hash=mycluster-767954bbfd mysql.oracle.com/cluster=mycluster mysql.oracle.com/cluster-role=PRIMARY mysql.oracle.com/instance-type=group-member statefulset.kubernetes.io/pod-name=mycluster-0 tier=mysql Annotations: kopf.zalando.org/last-handled-configuration: {"spec":{"volumes":[{"name":"datadir","persistentVolumeClaim":{"claimName":"datadir-mycluster-0"}},{"name":"mycnfdata","emptyDir":{}},{"na... mysql.oracle.com/fqdn-template: {service}.{namespace}.svc.{domain} mysql.oracle.com/membership-info: {"memberId": "fd265094-39ed-11ef-a354-a61ef5709950", "lastTransitionTime": "2024-07-04T10:25:41Z", "lastProbeTime": "2024-07-04T10:25:41Z"... Status: Running IP: 10.244.0.4 IPs: IP: 10.244.0.4 Controlled By: StatefulSet/mycluster Init Containers: fixdatadir: Container ID: docker://5624e37fdb14d70cb75a350b00de31de5b7345933ebc29a7ae18e0a48f0cad9e Image: container-registry.oracle.com/mysql/community-operator:9.0.0-2.2.0 Image ID: docker-pullable://container-registry.oracle.com/mysql/community-operator@sha256:1daaad9a004e1e94e4fed1c5b56074e295c507cc7ff51cf38e89e8edef0263f6 Port: <none> Host Port: <none> Command: bash -c chown 27:27 /var/lib/mysql && chmod 0700 /var/lib/mysql State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 04 Jul 2024 15:42:27 +0530 Finished: Thu, 04 Jul 2024 15:42:27 +0530 Ready: True Restart Count: 0 Environment: MYSQL_OPERATOR_K8S_CLUSTER_DOMAIN: cluster.local MYSQLSH_CREDENTIAL_STORE_SAVE_PASSWORDS: never Mounts: /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro) initconf: Container ID: docker://c92d8661be8fea881f052498f5d1947538b801f2a61cdc5dcac75527ec88c0e9 Image: container-registry.oracle.com/mysql/community-operator:9.0.0-2.2.0 Image ID: docker-pullable://container-registry.oracle.com/mysql/community-operator@sha256:1daaad9a004e1e94e4fed1c5b56074e295c507cc7ff51cf38e89e8edef0263f6 Port: <none> Host Port: <none> Command: mysqlsh --log-level=@INFO --pym mysqloperator init --pod-name $(POD_NAME) --pod-namespace $(POD_NAMESPACE) --datadir /var/lib/mysql State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 04 Jul 2024 15:42:30 +0530 Finished: Thu, 04 Jul 2024 15:42:40 +0530 Ready: True Restart Count: 0 Environment: POD_NAME: mycluster-0 (v1:metadata.name) POD_NAMESPACE: mycluster (v1:metadata.namespace) MYSQL_OPERATOR_K8S_CLUSTER_DOMAIN: cluster.local MYSQLSH_USER_CONFIG_HOME: /tmp MYSQLSH_CREDENTIAL_STORE_SAVE_PASSWORDS: never Mounts: /mnt/initconf from initconfdir (ro) /mnt/mycnfdata from mycnfdata (rw) /tmp from initconf-tmp (rw) /var/lib/mysql from datadir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro) initmysql: Container ID: docker://a2fced6c6c444a9951ba7b93c7b1e138b9c6efbdc4ab8a39192961aa4d659893 Image: container-registry.oracle.com/mysql/community-server:9.0.0 Image ID: docker-pullable://container-registry.oracle.com/mysql/community-server@sha256:0d6dfa57e104bddf057d5a3ebee8279dbc737cfc32208bbff29c3977a7b73ddb Port: <none> Host Port: <none> Args: mysqld --user=mysql State: Terminated Reason: Completed Exit Code: 0 Started: Thu, 04 Jul 2024 15:42:58 +0530 Finished: Thu, 04 Jul 2024 15:43:22 +0530 Ready: True Restart Count: 0 Environment: MYSQL_INITIALIZE_ONLY: 1 MYSQL_ROOT_PASSWORD: <set to the key 'rootPassword' in secret 'mypwds'> Optional: false MYSQLSH_USER_CONFIG_HOME: /tmp Mounts: /docker-entrypoint-initdb.d from mycnfdata (rw,path="docker-entrypoint-initdb.d") /etc/my.cnf from mycnfdata (rw,path="my.cnf") /etc/my.cnf.d from mycnfdata (rw,path="my.cnf.d") /tmp from initmysql-tmp (rw) /var/lib/mysql from datadir (rw) /var/lib/mysql-files from varlibmysqlfiles (rw) /var/run/mysqld from rundir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro) Containers: sidecar: Container ID: docker://083922497e4d1f05505e6563e2ac0c416c726195363438b8d9243701856fae2b Image: container-registry.oracle.com/mysql/community-operator:9.0.0-2.2.0 Image ID: docker-pullable://container-registry.oracle.com/mysql/community-operator@sha256:1daaad9a004e1e94e4fed1c5b56074e295c507cc7ff51cf38e89e8edef0263f6 Port: <none> Host Port: <none> Command: mysqlsh --pym mysqloperator sidecar --pod-name $(POD_NAME) --pod-namespace $(POD_NAMESPACE) --datadir /var/lib/mysql State: Running Started: Thu, 04 Jul 2024 15:43:24 +0530 Ready: True Restart Count: 0 Environment: POD_NAME: mycluster-0 (v1:metadata.name) POD_NAMESPACE: mycluster (v1:metadata.namespace) MYSQL_UNIX_PORT: /var/run/mysqld/mysql.sock MYSQLSH_USER_CONFIG_HOME: /mysqlsh MYSQL_OPERATOR_K8S_CLUSTER_DOMAIN: cluster.local MYSQLSH_CREDENTIAL_STORE_SAVE_PASSWORDS: never Mounts: /etc/my.cnf from mycnfdata (rw,path="my.cnf") /etc/my.cnf.d from mycnfdata (rw,path="my.cnf.d") /mysqlsh from shellhome (rw) /tmp from sidecar-tmp (rw) /var/run/mysqld from rundir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro) mysql: Container ID: docker://0eeb2af4b3ff5991fa16ff3deeb1238b2328ba1177c2d803c5a6c32bd8925b9a Image: container-registry.oracle.com/mysql/community-server:9.0.0 Image ID: docker-pullable://container-registry.oracle.com/mysql/community-server@sha256:0d6dfa57e104bddf057d5a3ebee8279dbc737cfc32208bbff29c3977a7b73ddb Ports: 3306/TCP, 33060/TCP, 33061/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Args: mysqld --user=mysql State: Running Started: Thu, 04 Jul 2024 15:43:24 +0530 Ready: True Restart Count: 0 Liveness: exec [/livenessprobe.sh] delay=15s timeout=1s period=15s #success=1 #failure=10 Readiness: exec [/readinessprobe.sh] delay=10s timeout=1s period=5s #success=1 #failure=10000 Startup: exec [/livenessprobe.sh 8] delay=5s timeout=1s period=3s #success=1 #failure=10000 Environment: MYSQL_UNIX_PORT: /var/run/mysqld/mysql.sock MYSQLSH_CREDENTIAL_STORE_SAVE_PASSWORDS: never Mounts: /etc/my.cnf from mycnfdata (rw,path="my.cnf") /etc/my.cnf.d from mycnfdata (rw,path="my.cnf.d") /livenessprobe.sh from initconfdir (rw,path="livenessprobe.sh") /readinessprobe.sh from initconfdir (rw,path="readinessprobe.sh") /tmp from mysql-tmp (rw) /var/lib/mysql from datadir (rw) /var/lib/mysql-files from varlibmysqlfiles (rw) /var/run/mysqld from rundir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hz6tn (ro) Readiness Gates: Type Status mysql.oracle.com/configured True mysql.oracle.com/ready True Conditions: Type Status mysql.oracle.com/ready True mysql.oracle.com/configured True PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: datadir: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: datadir-mycluster-0 ReadOnly: false mycnfdata: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> rundir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> varlibmysqlfiles: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> initconfdir: Type: ConfigMap (a volume populated by a ConfigMap) Name: mycluster-initconf Optional: false shellhome: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> initconf-tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> initmysql-tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> mysql-tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> sidecar-tmp: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-hz6tn: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none>