Description:
Hey!
I faced an issue with the MySQL Operator. I played with it a little bit and after some errors, I thought it might be a good idea to delete everything belonging. So I deleted the Instances, Routers and the Operator.
Then I wanted to start the Operator "from scratch". The problem is that the Operator doesn't start anymore. In the logs I found this:
[2023-05-17 12:04:24,729] kopf.activities.star [INFO ] MySQL Operator/operator.py=2.0.9 timestamp=2023-03-31T06:20:17 kopf=1.35.4 uid=2
[2023-05-17 12:04:24,741] kopf.activities.star [INFO ] KUBERNETES_VERSION =1.23
[2023-05-17 12:04:24,741] kopf.activities.star [INFO ] OPERATOR_VERSION =2.0.9
[2023-05-17 12:04:24,741] kopf.activities.star [INFO ] OPERATOR_EDITION =community
[2023-05-17 12:04:24,742] kopf.activities.star [INFO ] OPERATOR_EDITIONS =['community', 'enterprise']
[2023-05-17 12:04:24,742] kopf.activities.star [INFO ] SHELL_VERSION =8.0.33
[2023-05-17 12:04:24,742] kopf.activities.star [INFO ] DEFAULT_VERSION_TAG=8.0.33
[2023-05-17 12:04:24,742] kopf.activities.star [INFO ] SIDECAR_VERSION_TAG=8.0.33-2.0.9
[2023-05-17 12:04:24,742] kopf.activities.star [INFO ] DEFAULT_IMAGE_REPOSITORY =container-registry.oracle.com/mysql
[2023-05-17 12:04:24,747] kopf.activities.star [ERROR ] Activity 'on_startup' failed with an exception. Will retry.
Traceback (most recent call last):
File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
result = await invoke_handler(
File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
result = await invocation.invoke(
File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/invocation.py", line 139, in invoke
await asyncio.shield(future) # slightly expensive: creates tasks
File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/operator.py", line 38, in on_startup
operator_cluster.monitor_existing_clusters(logger)
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/operator_cluster.py", line 46, in monitor_existing_clusters
g_group_monitor.monitor_cluster(
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/group_monitor.py", line 202, in monitor_cluster
account = RetryLoop(logger).call(cluster.get_admin_account)
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/shellutils.py", line 93, in call
return f(*args)
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/cluster_api.py", line 1325, in get_admin_account
secrets = self.get_private_secrets()
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/cluster_api.py", line 1253, in get_private_secrets
api_core.read_namespaced_secret(f"{self.name}-privsecrets", self.namespace))
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/core_v1_api.py", line 24803, in read_namespaced_secret
return self.read_namespaced_secret_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/core_v1_api.py", line 24890, in read_namespaced_secret_with_http_info
return self.api_client.call_api(
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '677fc747-5591-461b-9c88-6b68514059cc', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '280523e8-eb1f-4e71-8c68-63a5e4cf193a', 'X-Kubernetes-Pf-Prioritylevel-Uid': '5719dc5e-db3a-4ee5-bbb3-4b1617c1f78c', 'Date': 'Wed, 17 May 2023 12:04:24 GMT', 'Content-Length': '240'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"mysql-test-db-cluster-privsecrets\" not found","reason":"NotFound","details":{"name":"mysql-test-db-cluster-privsecrets","kind":"secrets"},"code":404}
I also looked in the source code, but I can't break down the API call behind it. I doesn't understand, why the Operator searches for the mysql-test-db-cluster-privsecrets, although I deleted everything belonging to my previous experiment. Furthermore, I checked, that every PV, PVC, Pod, ... was deleted. But from where the Operator is getting the information, that an old secret has to exist?
How to repeat:
Create Operator and new InnoDBCluster.
Delete everything, then try to start a new Cluster.
Description: Hey! I faced an issue with the MySQL Operator. I played with it a little bit and after some errors, I thought it might be a good idea to delete everything belonging. So I deleted the Instances, Routers and the Operator. Then I wanted to start the Operator "from scratch". The problem is that the Operator doesn't start anymore. In the logs I found this: [2023-05-17 12:04:24,729] kopf.activities.star [INFO ] MySQL Operator/operator.py=2.0.9 timestamp=2023-03-31T06:20:17 kopf=1.35.4 uid=2 [2023-05-17 12:04:24,741] kopf.activities.star [INFO ] KUBERNETES_VERSION =1.23 [2023-05-17 12:04:24,741] kopf.activities.star [INFO ] OPERATOR_VERSION =2.0.9 [2023-05-17 12:04:24,741] kopf.activities.star [INFO ] OPERATOR_EDITION =community [2023-05-17 12:04:24,742] kopf.activities.star [INFO ] OPERATOR_EDITIONS =['community', 'enterprise'] [2023-05-17 12:04:24,742] kopf.activities.star [INFO ] SHELL_VERSION =8.0.33 [2023-05-17 12:04:24,742] kopf.activities.star [INFO ] DEFAULT_VERSION_TAG=8.0.33 [2023-05-17 12:04:24,742] kopf.activities.star [INFO ] SIDECAR_VERSION_TAG=8.0.33-2.0.9 [2023-05-17 12:04:24,742] kopf.activities.star [INFO ] DEFAULT_IMAGE_REPOSITORY =container-registry.oracle.com/mysql [2023-05-17 12:04:24,747] kopf.activities.star [ERROR ] Activity 'on_startup' failed with an exception. Will retry. Traceback (most recent call last): File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once result = await invoke_handler( File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler result = await invocation.invoke( File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/invocation.py", line 139, in invoke await asyncio.shield(future) # slightly expensive: creates tasks File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/operator.py", line 38, in on_startup operator_cluster.monitor_existing_clusters(logger) File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/operator_cluster.py", line 46, in monitor_existing_clusters g_group_monitor.monitor_cluster( File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/group_monitor.py", line 202, in monitor_cluster account = RetryLoop(logger).call(cluster.get_admin_account) File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/shellutils.py", line 93, in call return f(*args) File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/cluster_api.py", line 1325, in get_admin_account secrets = self.get_private_secrets() File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/cluster_api.py", line 1253, in get_private_secrets api_core.read_namespaced_secret(f"{self.name}-privsecrets", self.namespace)) File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/core_v1_api.py", line 24803, in read_namespaced_secret return self.read_namespaced_secret_with_http_info(name, namespace, **kwargs) # noqa: E501 File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/core_v1_api.py", line 24890, in read_namespaced_secret_with_http_info return self.api_client.call_api( File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 240, in GET return self.request("GET", url, File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 234, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (404) Reason: Not Found HTTP response headers: HTTPHeaderDict({'Audit-Id': '677fc747-5591-461b-9c88-6b68514059cc', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '280523e8-eb1f-4e71-8c68-63a5e4cf193a', 'X-Kubernetes-Pf-Prioritylevel-Uid': '5719dc5e-db3a-4ee5-bbb3-4b1617c1f78c', 'Date': 'Wed, 17 May 2023 12:04:24 GMT', 'Content-Length': '240'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"mysql-test-db-cluster-privsecrets\" not found","reason":"NotFound","details":{"name":"mysql-test-db-cluster-privsecrets","kind":"secrets"},"code":404} I also looked in the source code, but I can't break down the API call behind it. I doesn't understand, why the Operator searches for the mysql-test-db-cluster-privsecrets, although I deleted everything belonging to my previous experiment. Furthermore, I checked, that every PV, PVC, Pod, ... was deleted. But from where the Operator is getting the information, that an old secret has to exist? How to repeat: Create Operator and new InnoDBCluster. Delete everything, then try to start a new Cluster.