Bug #111817 MySQL Operator - stuck when InnoDBCluster object is deleted
Description:
Trying to use initDB object i deleted the object:
kubectl -n mysql delete InnoDBCluster mysql
innodbcluster.mysql.oracle.com "mysql" deleted
but this action got stuck.
In operator logs i can only see this printed in loop:
[2023-07-19 14:42:08,848] kopf.objects [INFO ] on_innodbcluster_field_backup_schedules
[2023-07-19 14:42:08,852] kopf.objects [ERROR ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' failed temporarily: The cluster is not ready. Will create the schedules once the first instance is up and running
[2023-07-19 14:42:18,997] kopf.objects [INFO ] on_innodbcluster_field_backup_schedules
[2023-07-19 14:42:19,000] kopf.objects [ERROR ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' failed temporarily: The cluster is not ready. Will create the schedules once the first instance is up and running
[2023-07-19 14:42:29,140] kopf.objects [INFO ] on_innodbcluster_field_backup_schedules
[2023-07-19 14:42:29,144] kopf.objects [ERROR ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' failed temporarily: The cluster is not ready. Will create the schedules once the first instance is up and running
InnoDBCluster object status
kubectl -n mysql get InnoDBCluster
NAME STATUS ONLINE INSTANCES ROUTERS AGE
mysql INITIALIZING 0 3 2 44m
Only operator pod is running:
mysql-operator-79d96c78c9-xn9qk 1/1 Running 0 21d
Now i m stuck, as i can't delete the InnoDBCluster and if i run helm upgrade nothing happens.
Thanks,
Carlos
How to repeat:
I have:
1-deploy cluster
2-added initDB values:
initDB:
dump:
name:
path: mysql-backup-schedule230719030102
# options:
# includeSchemas:
persistentVolumeClaim:
claimName: backup-volume-claim-1
3- I did helm upgrade to create InnoDBCluster again - Restore didn't worked
4- Deleted InnoDBCluster to see if create InnoDBCluster again would work
5- helm upgrade to create the InnoDBCluster object - Restore didn't worked
6- after troubleshooting the helm templates realised that name was needed (missing the documentation)
7- Deleted InnoDBCluster to see if create InnoDBCluster again would work
After this second delete got to that state.
Suggested fix:
Deleting a InnoDBCluster object should not get stuck like this, as explained now i can't delete or create the cluster again.
Also in https://dev.mysql.com/doc/mysql-operator/en/mysql-operator-properties.html#mysql-operator-... is state that name is not required (false) but seems wrong, looking to helm template:
..
{{- if and .name (or .ociObjectStorage .s3 .persistentVolumeClaim .options) }}
initDB:
dump:
..
it is needed, docs should be fixed required to true
Portions of this website are copyright © 2001, 2002 The PHP Group
Page generated in 0.052 sec. using MySQL 8.4.7-u6-cloud
Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the
respective copyright holders.
It is not reviewed in advance
by Oracle and does not necessarily represent the opinion of
Oracle or any other party.
Description: Trying to use initDB object i deleted the object: kubectl -n mysql delete InnoDBCluster mysql innodbcluster.mysql.oracle.com "mysql" deleted but this action got stuck. In operator logs i can only see this printed in loop: [2023-07-19 14:42:08,848] kopf.objects [INFO ] on_innodbcluster_field_backup_schedules [2023-07-19 14:42:08,852] kopf.objects [ERROR ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' failed temporarily: The cluster is not ready. Will create the schedules once the first instance is up and running [2023-07-19 14:42:18,997] kopf.objects [INFO ] on_innodbcluster_field_backup_schedules [2023-07-19 14:42:19,000] kopf.objects [ERROR ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' failed temporarily: The cluster is not ready. Will create the schedules once the first instance is up and running [2023-07-19 14:42:29,140] kopf.objects [INFO ] on_innodbcluster_field_backup_schedules [2023-07-19 14:42:29,144] kopf.objects [ERROR ] Handler 'on_innodbcluster_field_backup_schedules/spec.backupSchedules' failed temporarily: The cluster is not ready. Will create the schedules once the first instance is up and running InnoDBCluster object status kubectl -n mysql get InnoDBCluster NAME STATUS ONLINE INSTANCES ROUTERS AGE mysql INITIALIZING 0 3 2 44m Only operator pod is running: mysql-operator-79d96c78c9-xn9qk 1/1 Running 0 21d Now i m stuck, as i can't delete the InnoDBCluster and if i run helm upgrade nothing happens. Thanks, Carlos How to repeat: I have: 1-deploy cluster 2-added initDB values: initDB: dump: name: path: mysql-backup-schedule230719030102 # options: # includeSchemas: persistentVolumeClaim: claimName: backup-volume-claim-1 3- I did helm upgrade to create InnoDBCluster again - Restore didn't worked 4- Deleted InnoDBCluster to see if create InnoDBCluster again would work 5- helm upgrade to create the InnoDBCluster object - Restore didn't worked 6- after troubleshooting the helm templates realised that name was needed (missing the documentation) 7- Deleted InnoDBCluster to see if create InnoDBCluster again would work After this second delete got to that state. Suggested fix: Deleting a InnoDBCluster object should not get stuck like this, as explained now i can't delete or create the cluster again. Also in https://dev.mysql.com/doc/mysql-operator/en/mysql-operator-properties.html#mysql-operator-... is state that name is not required (false) but seems wrong, looking to helm template: .. {{- if and .name (or .ociObjectStorage .s3 .persistentVolumeClaim .options) }} initDB: dump: .. it is needed, docs should be fixed required to true