Bug #117277 Operator crashes when start up
Submitted: 23 Jan 7:47 Modified: 27 Jan 13:13
Reporter: Bing Ma (OCA) Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Operator Severity:S2 (Serious)
Version: OS:Any
Assigned to: MySQL Verification Team CPU Architecture:Any

[23 Jan 7:47] Bing Ma 
Description:
when start a operator, it panic
##############################################

[2025-01-23 07:42:28,336] kopf.activities.star [INFO    ] MySQL Operator/operator.py=2.2.2 timestamp=2024-10-11T06:34:35 kopf=1.35.4 uid=2
[2025-01-23 07:42:28,341] kopf.activities.star [INFO    ] KUBERNETES_VERSION =1.26
[2025-01-23 07:42:28,341] kopf.activities.star [INFO    ] OPERATOR_VERSION   =2.2.2
[2025-01-23 07:42:28,341] kopf.activities.star [INFO    ] OPERATOR_EDITION   =community
[2025-01-23 07:42:28,341] kopf.activities.star [INFO    ] OPERATOR_EDITIONS  =['community', 'enterprise']
[2025-01-23 07:42:28,341] kopf.activities.star [INFO    ] SHELL_VERSION      =9.1.0
[2025-01-23 07:42:28,342] kopf.activities.star [INFO    ] DEFAULT_VERSION_TAG=9.1.0
[2025-01-23 07:42:28,342] kopf.activities.star [INFO    ] SIDECAR_VERSION_TAG=9.1.0-2.2.2
[2025-01-23 07:42:28,342] kopf.activities.star [INFO    ] DEFAULT_IMAGE_REPOSITORY   =container-registry.oracle.com/mysql
[2025-01-23 07:42:28,343] kopf.activities.star [INFO    ] setuptools           = 50.3.2
[2025-01-23 07:42:28,343] kopf.activities.star [INFO    ] pip                  = 20.2.4
[2025-01-23 07:42:28,343] kopf.activities.star [INFO    ] yarl                 = 1.9.4
[2025-01-23 07:42:28,343] kopf.activities.star [INFO    ] websocket-client     = 1.7.0
[2025-01-23 07:42:28,343] kopf.activities.star [INFO    ] urllib3              = 2.2.0
[2025-01-23 07:42:28,343] kopf.activities.star [INFO    ] typing-extensions    = 4.9.0
[2025-01-23 07:42:28,343] kopf.activities.star [INFO    ] six                  = 1.16.0
[2025-01-23 07:42:28,344] kopf.activities.star [INFO    ] rsa                  = 4.9
[2025-01-23 07:42:28,344] kopf.activities.star [INFO    ] requests             = 2.31.0
[2025-01-23 07:42:28,344] kopf.activities.star [INFO    ] requests-oauthlib    = 1.3.1
[2025-01-23 07:42:28,344] kopf.activities.star [INFO    ] PyYAML               = 6.0.1
[2025-01-23 07:42:28,344] kopf.activities.star [INFO    ] python-json-logger   = 2.0.7
[2025-01-23 07:42:28,344] kopf.activities.star [INFO    ] python-dateutil      = 2.8.2
[2025-01-23 07:42:28,344] kopf.activities.star [INFO    ] pyasn1               = 0.5.1
[2025-01-23 07:42:28,345] kopf.activities.star [INFO    ] pyasn1-modules       = 0.3.0
[2025-01-23 07:42:28,345] kopf.activities.star [INFO    ] oauthlib             = 3.2.2
[2025-01-23 07:42:28,345] kopf.activities.star [INFO    ] multidict            = 6.0.5
[2025-01-23 07:42:28,345] kopf.activities.star [INFO    ] kubernetes           = 23.6.0
[2025-01-23 07:42:28,345] kopf.activities.star [INFO    ] kopf                 = 1.35.4
[2025-01-23 07:42:28,346] kopf.activities.star [INFO    ] iso8601              = 2.1.0
[2025-01-23 07:42:28,346] kopf.activities.star [INFO    ] idna                 = 3.6
[2025-01-23 07:42:28,346] kopf.activities.star [INFO    ] google-auth          = 2.27.0
[2025-01-23 07:42:28,346] kopf.activities.star [INFO    ] frozenlist           = 1.4.1
[2025-01-23 07:42:28,346] kopf.activities.star [INFO    ] click                = 8.1.7
[2025-01-23 07:42:28,346] kopf.activities.star [INFO    ] charset-normalizer   = 3.3.2
[2025-01-23 07:42:28,347] kopf.activities.star [INFO    ] certifi              = 2024.2.2
[2025-01-23 07:42:28,347] kopf.activities.star [INFO    ] cachetools           = 5.3.2
[2025-01-23 07:42:28,347] kopf.activities.star [INFO    ] attrs                = 23.2.0
[2025-01-23 07:42:28,347] kopf.activities.star [INFO    ] async-timeout        = 4.0.3
[2025-01-23 07:42:28,348] kopf.activities.star [INFO    ] aiosignal            = 1.3.1
[2025-01-23 07:42:28,348] kopf.activities.star [INFO    ] aiohttp              = 3.8.5
[2025-01-23 07:42:28,348] kopf.activities.star [INFO    ] pytz                 = 2024.1
[2025-01-23 07:42:28,348] kopf.activities.star [INFO    ] pyOpenSSL            = 24.2.1
[2025-01-23 07:42:28,348] kopf.activities.star [INFO    ] PyNaCl               = 1.5.0
[2025-01-23 07:42:28,348] kopf.activities.star [INFO    ] pycparser            = 2.22
[2025-01-23 07:42:28,349] kopf.activities.star [INFO    ] paramiko             = 3.4.0
[2025-01-23 07:42:28,349] kopf.activities.star [INFO    ] oci                  = 2.133.0
[2025-01-23 07:42:28,349] kopf.activities.star [INFO    ] cryptography         = 42.0.8
[2025-01-23 07:42:28,349] kopf.activities.star [INFO    ] circuitbreaker       = 2.0.0
[2025-01-23 07:42:28,350] kopf.activities.star [INFO    ] cffi                 = 1.17.0
[2025-01-23 07:42:28,350] kopf.activities.star [INFO    ] bcrypt               = 4.1.3
[2025-01-23 07:42:28,350] kopf.activities.star [INFO    ] antlr4-python3-runtime = 4.13.1
[2025-01-23 07:42:28,634] kopf.activities.star [ERROR   ] Activity 'on_startup' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
    result = await invoke_handler(
  File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
    result = await invocation.invoke(
  File "/usr/lib/mysqlsh/python-packages/kopf/_core/actions/invocation.py", line 139, in invoke
    await asyncio.shield(future)  # slightly expensive: creates tasks
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/operator.py", line 45, in on_startup
    operator_cluster.ensure_router_accounts_are_uptodate(clusters, logger)
  File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/operator_cluster.py", line 62, in ensure_router_accounts_are_uptodate
    router_objects.update_router_account(cluster,
  File "/usr/lib/mysqlsh/python-packages/mysqloperator/controller/innodbcluster/router_objects.py", line 481, in update_router_account
    dba.get_cluster().setup_router_account(user, {"update": True})
RuntimeError: This function is not available through a session to an instance belonging to an unmanaged replication group

How to repeat:
I don't know how to repeat, the only instance looks good except one node is Recovering 

 MySQL  localhost:33060+ ssl  test  JS > cluster.status()
{
    "clusterName": "ic_new_op_0120",
    "defaultReplicaSet": {
        "name": "default",
        "primary": "ic-new-op-0120-2.ic-new-op-0120-instances.default.svc.cluster.local:3306",
        "ssl": "REQUIRED",
        "status": "OK",
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure. 1 member is not active.",
        "topology": {
            "ic-new-op-0120-0.ic-new-op-0120-instances.default.svc.cluster.local:3306": {
                "address": "ic-new-op-0120-0.ic-new-op-0120-instances.default.svc.cluster.local:3306",
                "memberRole": "SECONDARY",
                "mode": "n/a",
                "readReplicas": {},
                "recovery": {
                    "state": "ON"
                },
                "recoveryStatusText": "Distributed recovery in progress",
                "role": "HA",
                "status": "RECOVERING",
                "version": "8.0.37"
            },
            "ic-new-op-0120-1.ic-new-op-0120-instances.default.svc.cluster.local:3306": {
                "address": "ic-new-op-0120-1.ic-new-op-0120-instances.default.svc.cluster.local:3306",
                "memberRole": "SECONDARY",
                "mode": "R/O",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.0.37"
            },
            "ic-new-op-0120-2.ic-new-op-0120-instances.default.svc.cluster.local:3306": {
                "address": "ic-new-op-0120-2.ic-new-op-0120-instances.default.svc.cluster.local:3306",
                "memberRole": "PRIMARY",
                "mode": "R/W",
                "readReplicas": {},
                "replicationLag": "applier_queue_applied",
                "role": "HA",
                "status": "ONLINE",
                "version": "8.0.37"
            }
        },
        "topologyMode": "Single-Primary"
    },
    "groupInformationSourceMember": "ic-new-op-0120-2.ic-new-op-0120-instances.default.svc.cluster.local:3306"
}
[27 Jan 13:13] MySQL Verification Team 
Hi,

Thank you for the report but I cannot reproduce this.
Can you please upgrade (and retry) to 8.0.41 as 8.0.37 is a year old.

Thanks