| Bug #110447 | Not able to create innodb cluster | ||
|---|---|---|---|
| Submitted: | 21 Mar 2023 4:45 | Modified: | 21 Apr 2023 13:28 |
| Reporter: | Tanu Ajmera | Email Updates: | |
| Status: | No Feedback | Impact on me: | |
| Category: | MySQL Operator | Severity: | S2 (Serious) |
| Version: | 8.0.32-2.0.8 | OS: | Any |
| Assigned to: | MySQL Verification Team | CPU Architecture: | Any |
[21 Mar 2023 13:28]
MySQL Verification Team
Hi, I just tried this on my test setup and it all went ok. I am unable to reproduce this problem. Can you please give us more detailed steps to reproduce the problem.
[22 Apr 2023 1:00]
Bugs System
No feedback was provided for this bug for over a month, so it is being suspended automatically. If you are able to provide the information that was originally requested, please do so and change the status of the bug back to "Open".
[19 Jun 2023 20:31]
Aikoroje Imonikhe
I am faceing similar issues too
mysql-operator 0/1 1 0 79s
kubectl get logs mysql-operator-hdhdhdd-dddd -n mysql-operator
[2023-06-19 20:28:50,192] kopf.activities.star [INFO ] MySQL Operator/operator.py=2.0.10 timestamp=2023-05-09T13:27:59 kopf=1.35.4 uid=2
[2023-06-19 20:28:50,194] kopf.activities.star [INFO ] KUBERNETES_VERSION =1.25
[2023-06-19 20:28:50,194] kopf.activities.star [INFO ] OPERATOR_VERSION =2.0.10
[2023-06-19 20:28:50,194] kopf.activities.star [INFO ] OPERATOR_EDITION =community
[2023-06-19 20:28:50,195] kopf.activities.star [INFO ] OPERATOR_EDITIONS =['community', 'enterprise']
[2023-06-19 20:28:50,195] kopf.activities.star [INFO ] SHELL_VERSION =8.0.33
[2023-06-19 20:28:50,195] kopf.activities.star [INFO ] DEFAULT_VERSION_TAG=8.0.33
[2023-06-19 20:28:50,195] kopf.activities.star [INFO ] SIDECAR_VERSION_TAG=8.0.33-2.0.10
[2023-06-19 20:28:50,195] kopf.activities.star [INFO ] DEFAULT_IMAGE_REPOSITORY =container-registry.oracle.com/mysql
[2023-06-19 20:28:50,206] kopf.activities.star [ERROR ] Activity 'on_startup' failed with an exception. Will retry.
Traceback (most recent call last):
File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
result = await invoke_handler(
File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
result = await invocation.invoke(
File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/invocation.py", line 139, in invoke
await asyncio.shield(future) # slightly expensive: creates tasks
File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/operator.py", line 38, in on_startup
operator_cluster.monitor_existing_clusters(logger)
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/operator_cluster.py", line 46, in monitor_existing_clusters
g_group_monitor.monitor_cluster(
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/group_monitor.py", line 202, in monitor_cluster
account = RetryLoop(logger).call(cluster.get_admin_account)
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/shellutils.py", line 93, in call
return f(*args)
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/cluster_api.py", line 1325, in get_admin_account
secrets = self.get_private_secrets()
File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/cluster_api.py", line 1253, in get_private_secrets
api_core.read_namespaced_secret(f"{self.name}-privsecrets", self.namespace))
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/core_v1_api.py", line 24803, in read_namespaced_secret
return self.read_namespaced_secret_with_http_info(name, namespace, **kwargs) # noqa: E501
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/core_v1_api.py", line 24890, in read_namespaced_secret_with_http_info
return self.api_client.call_api(
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e04a0113-cb32-441a-bf4f-37d1a4ebc914', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '736927a7-08de-434f-bb5f-f62e9932e45e', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a4151359-83ce-4eea-b8c2-12f2caa834bc', 'Date': 'Mon, 19 Jun 2023 20:28:50 GMT', 'Content-Length': '208'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"mycls-privsecrets\" not found","reason":"NotFound","details":{"name":"mycls-privsecrets","kind":"secrets"},"code":404}
Description: File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/invocation.py", line 139, in invoke await asyncio.shield(future) # slightly expensive: creates tasks File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/operator.py", line 36, in on_startup operator_cluster.monitor_existing_clusters(logger) File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/operator_cluster.py", line 43, in monitor_existing_clusters clusters = cluster_api.get_all_clusters() File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/cluster_api.py", line 1556, in get_all_clusters objects = cast(dict, api_customobj.list_cluster_custom_object( File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/custom_objects_api.py", line 1942, in list_cluster_custom_object return self.list_cluster_custom_object_with_http_info(group, version, plural, **kwargs) # noqa: E501 File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api/custom_objects_api.py", line 2073, in list_cluster_custom_object_with_http_info return self.api_client.call_api( File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( File "/usr/lib/mysqlsh/python-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 240, in GET return self.request("GET", url, File "/usr/lib/mysqlsh/python-packages/kubernetes/client/rest.py", line 234, in request raise ApiException(http_resp=r) kubernetes.client.exceptions.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f4567936-0266-4f03-be81-0f0b75637a14', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'f5b32142-7e81-40e1-a1ec-ab3a92923525', 'X-Kubernetes-Pf-Prioritylevel-Uid': '828f8164-080f-4dde-86c5-f60bf35ce827', 'Date': 'Tue, 21 Mar 2023 04:39:11 GMT', 'Content-Length': '372'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"innodbclusters.mysql.oracle.com is forbidden: User \"system:serviceaccount:tanu1:mysql-operator-sa\" cannot list resource \"innodbclusters\" in API group \"mysql.oracle.com\" at the cluster scope","reason":"Forbidden","details":{"group":"mysql.oracle.com","kind":"innodbclusters"},"code":403} mysql-operator shows the above error when trying to install. This is occured because there are some innodb cluster instances stuck in terminating state due to finalizers not removed properly which doesn't let mysql-operator to connect. This creates problem in installing new innodb cluster How to repeat: Create an innodb cluster in namespace1. Add finalizers section and try to delete the pod. The pod will be stuck in terminating state. Now try to create another innodb cluster in namespace2. You will see no cluster come up. If you check mysql-operator logs, you can find the above error trace.