Description:
sometimes the logs of sidecar container reports:
kopf._cogs.clients.errors.APIForbiddenError: ('secrets is forbidden: User "system:serviceaccount:m0103:mgr0416-sidecar-sa" cannot watch resource "secrets" in API group "" in the namespace "m0103": RBAC: clusterrole.rbac.authorization.k8s.io "mysql-sidecar" not found', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets is forbidden: User "system:serviceaccount:m0103:mgr0416-sidecar-sa" cannot watch resource "secrets" in API group "" in the namespace "m0103": RBAC: clusterrole.rbac.authorization.k8s.io "mysql-sidecar" not found', 'reason': 'Forbidden', 'details': {'kind': 'secrets'}, 'code': 403})
####
after restart the pod, it missed. but some warnings and some similar errors:
[2024-04-07 13:55:41,660] kopf._core.reactor.o [WARNING ] Not enough permissions to list namespaces. Falling back to a list of namespaces which are assumed to exist: {'mcamel-system'}
[2024-04-07 13:55:41,673] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for resources: changes (creation/deletion/updates) will not be noticed; the resources are only refreshed on operator restarts.
[2024-04-07 13:55:41,674] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for namespaces: changes (deletion/creation) will not be noticed; the namespaces are only refreshed on operator restarts.
[2024-04-11 14:23:25,967] kopf._cogs.clients.w [ERROR ] Request attempt #1/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-11 14:23:47,039] kopf._cogs.clients.w [ERROR ] Request attempt #2/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-11 14:23:48,043] kopf._cogs.clients.w [ERROR ] Request attempt #3/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-11 14:23:50,047] kopf._cogs.clients.w [ERROR ] Request attempt #4/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-11 14:23:53,050] kopf._cogs.clients.w [ERROR ] Request attempt #5/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-11 14:23:58,057] kopf._cogs.clients.w [ERROR ] Request attempt #6/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-11 14:24:06,068] kopf._cogs.clients.w [ERROR ] Request attempt #7/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-11 14:24:19,083] kopf._cogs.clients.w [ERROR ] Request attempt #8/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-14 08:20:41,837] kopf._cogs.clients.w [ERROR ] Request attempt #1/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-14 08:20:49,242] kopf._cogs.clients.w [ERROR ] Request attempt #2/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
[2024-04-14 08:22:40,012] urllib3.connectionpo [WARNING ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f10cc5a3e80>: Failed to establish a new connection: [Errno 111] Connection refused')': /apis/mysql.oracle.com/v2/namespaces/mcamel-system/innodbclusters/kpanda-mgr
[2024-04-18 12:41:35,834] kopf._cogs.clients.w [ERROR ] Request attempt #1/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)"))
How to repeat:
I don't know how to repeat this,but I also noticed the TODO in source code:
# TODO - create ServiceAccount ({cluster.name}-sidecar-sa) for the mysql pods and bind it to the mysql-sidecar role
Description: sometimes the logs of sidecar container reports: kopf._cogs.clients.errors.APIForbiddenError: ('secrets is forbidden: User "system:serviceaccount:m0103:mgr0416-sidecar-sa" cannot watch resource "secrets" in API group "" in the namespace "m0103": RBAC: clusterrole.rbac.authorization.k8s.io "mysql-sidecar" not found', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets is forbidden: User "system:serviceaccount:m0103:mgr0416-sidecar-sa" cannot watch resource "secrets" in API group "" in the namespace "m0103": RBAC: clusterrole.rbac.authorization.k8s.io "mysql-sidecar" not found', 'reason': 'Forbidden', 'details': {'kind': 'secrets'}, 'code': 403}) #### after restart the pod, it missed. but some warnings and some similar errors: [2024-04-07 13:55:41,660] kopf._core.reactor.o [WARNING ] Not enough permissions to list namespaces. Falling back to a list of namespaces which are assumed to exist: {'mcamel-system'} [2024-04-07 13:55:41,673] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for resources: changes (creation/deletion/updates) will not be noticed; the resources are only refreshed on operator restarts. [2024-04-07 13:55:41,674] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for namespaces: changes (deletion/creation) will not be noticed; the namespaces are only refreshed on operator restarts. [2024-04-11 14:23:25,967] kopf._cogs.clients.w [ERROR ] Request attempt #1/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-11 14:23:47,039] kopf._cogs.clients.w [ERROR ] Request attempt #2/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-11 14:23:48,043] kopf._cogs.clients.w [ERROR ] Request attempt #3/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-11 14:23:50,047] kopf._cogs.clients.w [ERROR ] Request attempt #4/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-11 14:23:53,050] kopf._cogs.clients.w [ERROR ] Request attempt #5/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-11 14:23:58,057] kopf._cogs.clients.w [ERROR ] Request attempt #6/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-11 14:24:06,068] kopf._cogs.clients.w [ERROR ] Request attempt #7/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-11 14:24:19,083] kopf._cogs.clients.w [ERROR ] Request attempt #8/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-14 08:20:41,837] kopf._cogs.clients.w [ERROR ] Request attempt #1/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-14 08:20:49,242] kopf._cogs.clients.w [ERROR ] Request attempt #2/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) [2024-04-14 08:22:40,012] urllib3.connectionpo [WARNING ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f10cc5a3e80>: Failed to establish a new connection: [Errno 111] Connection refused')': /apis/mysql.oracle.com/v2/namespaces/mcamel-system/innodbclusters/kpanda-mgr [2024-04-18 12:41:35,834] kopf._cogs.clients.w [ERROR ] Request attempt #1/9 failed; will retry: GET https://10.233.0.1:443/api/v1/namespaces/mcamel-system/secrets?watch=true&resourceVersion=... -> ClientConnectorError(ConnectionKey(host='10.233.0.1', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=-1320582141914143285), ConnectionRefusedError(111, "Connect call failed ('10.233.0.1', 443)")) How to repeat: I don't know how to repeat this,but I also noticed the TODO in source code: # TODO - create ServiceAccount ({cluster.name}-sidecar-sa) for the mysql pods and bind it to the mysql-sidecar role