Bug #114893 Finalizer finalization process for the MySQL cluster got interrupted or stuck
Submitted: 6 May 2024 8:14 Modified: 20 Jan 3:25
Reporter: Ram Vibhul (OCA) Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Operator Severity:S1 (Critical)
Version:9.1.0-2.2.2 OS:Any
Assigned to: CPU Architecture:Any

[6 May 2024 8:14] Ram Vibhul
Description:
When you delete the node as part of the cluster-wide reboot to install OS updates, it seems that the finalization process for the MySQL cluster got interrupted or stuck. 
This interruption likely occurred because the finalizer could not complete its finalization tasks and as a result, the finalizer crashed and never recovered, leading to downtime because the mysql pod is not alive until you patch the finalizer with empty data and by forcefully removing other finalizers the pods comes back again.
Mysql server database is critical component and it needs to be more resilient when node deletion/drain happens

How to repeat:
1. Install the mysql operator
2. Deploy a simple MySQL cluster in a single node cluster with the following values:

credentials:
  root:
    user: root
    password: password
    host: "%"
tls:
  useSelfSigned: true
serverInstances: 3
routerInstances: 1

3. Delete a node in k8s cluster

4. When the new node boots up, the the pods gets stuck in Termination state forever.

Suggested fix:
Make the finalizer to handle the ungraceful termination of mysql pods.
[7 May 2024 7:26] MySQL Verification Team
Thank you for the report
[16 May 2024 7:30] Eric Trinh
Maybe the same as Bug #115014 ?
[17 May 2024 20:44] MySQL Verification Team
Bug #115014 is marked as duplicate of this one
[20 Jan 3:25] Ram Vibhul
Do we have any update on this? Why is it ignored though its marked as critical?