Bug #111817 MySQL Operator - stuck when InnoDBCluster object is deleted

Submitted: 19 Jul 2023 14:58 Modified: 21 Jul 2023 15:57
Reporter: Carlos Abrantes Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Operator Severity:S3 (Non-critical)
Version: OS:Any
Assigned to: CPU Architecture:Any

[19 Jul 2023 14:58] Carlos Abrantes
Description:
Trying to use initDB object i deleted the object:

kubectl -n mysql delete InnoDBCluster mysql
innodbcluster.mysql.oracle.com "mysql" deleted

but this action got stuck.

In operator logs i can only see this printed in loop:

[2023-07-19 14:42:08,848] kopf.objects         [INFO    ] on_innodbcluster_field_backup_schedules
[2023-07-19 14:42:08,852] kopf.objects         [ERROR   ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' failed temporarily: The cluster is not ready. Will create the schedules once the first instance is up and running
[2023-07-19 14:42:18,997] kopf.objects         [INFO    ] on_innodbcluster_field_backup_schedules
[2023-07-19 14:42:19,000] kopf.objects         [ERROR   ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' failed temporarily: The cluster is not ready. Will create the schedules once the first instance is up and running
[2023-07-19 14:42:29,140] kopf.objects         [INFO    ] on_innodbcluster_field_backup_schedules
[2023-07-19 14:42:29,144] kopf.objects         [ERROR   ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' failed temporarily: The cluster is not ready. Will create the schedules once the first instance is up and running

InnoDBCluster object status
kubectl -n mysql get InnoDBCluster
NAME    STATUS         ONLINE   INSTANCES   ROUTERS   AGE
mysql   INITIALIZING   0        3           2         44m

Only operator pod is running:
mysql-operator-79d96c78c9-xn9qk   1/1     Running   0          21d

Now i m stuck, as i can't delete the InnoDBCluster and if i run helm upgrade nothing happens.

Thanks,
Carlos

How to repeat:
I have:

1-deploy cluster

2-added initDB values:
initDB:
  dump:
    name:
    path: mysql-backup-schedule230719030102
#    options:
#      includeSchemas:
    persistentVolumeClaim:
        claimName: backup-volume-claim-1

3- I did helm upgrade to create InnoDBCluster again - Restore didn't worked

4- Deleted InnoDBCluster to see if create InnoDBCluster again would work

5- helm upgrade to create the InnoDBCluster object - Restore didn't worked

6- after troubleshooting the helm templates realised that name was needed (missing the documentation)

7- Deleted InnoDBCluster to see if create InnoDBCluster again would work

After this second delete got to that state.

Suggested fix:

Deleting a InnoDBCluster object should not get stuck like this, as explained now i  can't delete or create the cluster again.

Also in https://dev.mysql.com/doc/mysql-operator/en/mysql-operator-properties.html#mysql-operator-... is state that name is not required (false) but seems wrong, looking to helm template:

..
    {{- if and .name (or .ociObjectStorage .s3 .persistentVolumeClaim .options) }}
  initDB:
    dump:
..
it is needed, docs should be fixed required to true
[19 Jul 2023 14:58] Carlos Abrantes
Version:
community-operator:8.0.33-2.0.10
community-server:8.0.33
community-router:8.0.33

apiVersion: v2
appVersion: 8.0.33
description: MySQL InnoDB Cluster Helm Chart for deploying MySQL InnoDB Cluster in Kubernetes
icon: https://labs.mysql.com/common/themes/sakila/favicon.ico
name: mysql-innodbcluster
type: application
version: 2.0.10
[19 Jul 2023 17:07] Carlos Abrantes
Update:

I was only able to delete InnoDBCluster when i manually edit the InnoDBCluster removing the backup scheduler.
[21 Jul 2023 15:57] MySQL Verification Team
Thank you for your report

Portions of this website are copyright © 2001, 2002 The PHP Group

Page generated in 0.052 sec. using MySQL 8.4.7-u6-cloud

Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the respective copyright holders.
It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.