Bug #115438 the pod alway terminating

• Submitted: 27 Jun 2:53
• Modified: 1 Jul 8:05
• Reporter: Bing Ma (OCA)
• Status: Can't repeat
• Category: MySQL Operator
• Severity: S3 (Non-critical)
• OS: Linux
• Assigned to: MySQL Verification Team
• CPU Architecture: x86

[27 Jun 2:53] Bing Ma

Description:
until I manually deleted the pod's finalizer

How to repeat:
like power off or other things

[27 Jun 2:56] Bing Ma

We had a dns failuer, the operator log

[2024-06-27 02:42:56,762] kopf.objects         [INFO    ] Could not connect to mcamel-common-instance-name-1.mcamel-common-instance-name-instances.mcamel-system.svc.cluster.local:3306: error=MySQL Error (2005): mysqlsh.connect_dba: Unknown MySQL server host 'mcamel-common-instance-name-1.mcamel-common-instance-name-instances.mcamel-system.svc.cluster.local' (-2)
[2024-06-27 02:42:56,800] kopf.objects         [INFO    ] mcamel-common-instance-name-1.mcamel-common-instance-name-instances.mcamel-system.svc.cluster.local:3306: pod.phase=Failed  deleting=True
[2024-06-27 02:42:56,804] kopf.objects         [INFO    ] diag instance mcamel-common-instance-name-1 --> InstanceDiagStatus.OFFLINE quorum=None gtid_executed=None
[2024-06-27 02:42:56,827] kopf.objects         [INFO    ] Could not connect to mcamel-common-instance-name-2.mcamel-common-instance-name-instances.mcamel-system.svc.cluster.local:3306: error=MySQL Error (2005): mysqlsh.connect_dba: Unknown MySQL server host 'mcamel-common-instance-name-2.mcamel-common-instance-name-instances.mcamel-system.svc.cluster.local' (-2)
[2024-06-27 02:42:56,867] kopf.objects         [INFO    ] mcamel-common-instance-name-2.mcamel-common-instance-name-instances.mcamel-system.svc.cluster.local:3306: pod.phase=Failed  deleting=True
[2024-06-27 02:42:56,870] kopf.objects         [INFO    ] diag instance mcamel-common-instance-name-2 --> InstanceDiagStatus.OFFLINE quorum=None gtid_executed=None
[2024-06-27 02:42:56,901] kopf.objects         [INFO    ] Could not connect to mcamel-common-instance-name-0.mcamel-common-instance-name-instances.mcamel-system.svc.cluster.local:3306: error=MySQL Error (2005): mysqlsh.connect_dba: Unknown MySQL server host 'mcamel-common-instance-name-0.mcamel-common-instance-name-instances.mcamel-system.svc.cluster.local' (-2)
[2024-06-27 02:42:56,934] kopf.objects         [INFO    ] mcamel-common-instance-name-0.mcamel-common-instance-name-instances.mcamel-system.svc.cluster.local:3306: pod.phase=Pending  deleting=False
[2024-06-27 02:42:56,937] kopf.objects         [INFO    ] diag instance mcamel-common-instance-name-0 --> InstanceDiagStatus.OFFLINE quorum=None gtid_executed=None
[2024-06-27 02:42:56,937] kopf.objects         [INFO    ] mcamel-common-instance-name: all={<MySQLPod mcamel-common-instance-name-1>, <MySQLPod mcamel-common-instance-name-2>, <MySQLPod mcamel-common-instance-name-0>}  members={<MySQLPod mcamel-common-instance-name-1>, <MySQLPod mcamel-common-instance-name-2>, <MySQLPod mcamel-common-instance-name-0>}  online=set()  offline={<MySQLPod mcamel-common-instance-name-1>, <MySQLPod mcamel-common-instance-name-2>, <MySQLPod mcamel-common-instance-name-0>}  unsure=set()
[2024-06-27 02:42:57,033] kopf.objects         [INFO    ] cluster probe: status=ClusterDiagStatus.OFFLINE online=[]
[2024-06-27 02:42:57,036] kopf.objects         [INFO    ] ATTEMPTING CLUSTER REPAIR

[28 Jun 15:51] MySQL Verification Team

Hi,

I was not able to reproduce this? 

Can you give us more data on how to reproduce?

Thanks

[1 Jul 8:05] Bing Ma

the operator reports: Handler 'on_pod_delete' failed temporarily: Cluster cannot be restored because there are unreachable pods