Bug #121670 Operator/Router can't connect when clients exhaust max_connections

Submitted: 5 Nov 2:23
Reporter: minkwan cho Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Operator Severity:S2 (Serious)
Version:26.7.0-2.3.0 (Server 9.7.1, Router 26.7. OS:Ubuntu (22.04)
Assigned to: CPU Architecture:Any
Tags: admin_port, Error 1040, max_connections, metadata_cache, mysql-operator

[5 Nov 2:23] minkwan cho
Description:
MySQL Operator and the MySQL Router it deploys reach every InnoDB Cluster member
only through the regular client port (3306). They share max_connections with all
application traffic. When client connections fill max_connections, the operator
and the routers can no longer open sessions. Automatic recovery stops, and after
a primary failure writes do not resume until clients release connections.

Details:
- The operator hard-codes the member port: MySQLPod.port = 3306
  (mysqloperator/controller/innodbcluster/cluster_api.py:3042). The
  InnoDBCluster CR has no field to use the administrative interface
  (admin_address/admin_port), although MySQL provides that interface for this
  exact situation.
- The server allows only ONE extra connection beyond max_connections for
  CONNECTION_ADMIN accounts. All of these accounts share it: localroot (sidecar),
  root, mysqladmin-* (operator), mysql_innodb_cluster_* (recovery). The router
  account (mysqlrouter-*) and mysqlbackup do not have CONNECTION_ADMIN.
- In steady state the internal accounts hold about 12 connections per member
  (operator 6, router 2, recovery 1-2, event_scheduler 1). A single
  rejoin/diagnose cycle opens several more.

Observed (max_connections=40, application account without admin privileges,
connections held through the router):
- Secondary mysqld killed: GR continued with 2 members. The operator failed to
  connect to the remaining members with "1040 Too many connections", left the
  instance in diagnose state OFFLINE_UNCERTAIN and retried every 60 s. No rejoin
  for 8+ minutes. After the client connections were released, the instance
  rejoined 70 s later.
- Primary mysqld killed: GR elected a new primary. However, all routers got 1040
  from all 3 members and logged "Failed fetching metadata from any of the 3
  metadata servers". The RW port did not follow the new primary, and writes were
  impossible for 8+ minutes. Recovery came 72 s after the client connections were
  released.
- Saturation alone, without a failure, did not show errors within 3 minutes. The
  operator only opens SQL sessions when it reconciles, so the problem appears
  when a failure coincides with a connection spike. Failovers themselves often
  cause such spikes.

Workaround attempts that failed:
- Swapping ports (port=3336 for clients, admin_port=3306 + admin_address=<pod
  FQDN>, so the operator's 3306 hits the admin interface): cluster creation
  never finishes (InnoDBCluster stays INITIALIZING, 0 members). The operator
  matches members by the string "<fqdn>:3306" (pod.endpoint) in
  cluster.status(). AdminAPI records the members as "<fqdn>:@@port" (:3336), so
  the operator logs "Could not find <pod> in local cluster.status() output"
  forever. Setting report_port=3306 instead would make Router send application
  traffic to the admin port, which is not usable either.
- The only remaining mitigation is to keep total client connections well below
  max_connections (per-account MAX_USER_CONNECTIONS, router
  max_total_connections). With 10 spare slots we still saw intermittent 1040 for
  router/operator. With 20 spare slots recovery was normal (router failover 22 s,
  rejoin about 1 min 50 s).

Related: mysql/mysql-operator GitHub issue #69 (router metadata refresh blocked
by 1040 on the new primary after failover). Bug #99917 (admin interface
connection limits).

How to repeat:
1. Deploy operator 26.7.0-2.3.0 and a 3-member InnoDBCluster (server 9.7.1,
   router 26.7.1, 2 router instances).
2. On every member: SET PERSIST max_connections = 40;
3. Create an application account without CONNECTION_ADMIN /
   SERVICE_CONNECTION_ADMIN:
     CREATE USER 'cust'@'%' IDENTIFIED BY '...';
     GRANT SELECT, INSERT, UPDATE, DELETE ON test.* TO 'cust'@'%';
4. From a client pod, open as many connections as possible as 'cust' through
   the router: RW port 6446 (~60) and RO port 6447 (~120). Keep them idle and
   reopen any dropped connection every second, so that every member stays at
   max_connections.
5. Secondary case: kill -9 mysqld in a SECONDARY pod.
   Watch the operator log (kubectl logs deploy/mysql-operator) and
   performance_schema.replication_group_members.
   -> operator: "1040 Too many connections", diagnose OFFLINE_UNCERTAIN, the
      member does not rejoin while the client connections are held.
6. Release the client connections -> the member rejoins about 70 s later.
7. Primary case: repeat step 4, then kill -9 mysqld in the PRIMARY pod.
   -> router log: "Failed fetching metadata from any of the 3 metadata
      servers"; writes through 6446 keep failing; no rejoin.
8. Release the client connections -> service recovers about 72 s later.

Suggested fix:
1. Allow the operator's control sessions to use the administrative interface.
   For example, add a CR option that enables admin_address=<pod FQDN/IP> and
   admin_port per member. The operator then connects to that port.
   mysqladmin-* already has SERVICE_CONNECTION_ADMIN via GRANT ALL.
2. Separate "member identity" (the metadata address <report_host>:<port>) from
   the "operator connect port" in MySQLPod.endpoint / endpoint_co and in
   diagnose.py. Changing the connect port must not break member matching
   against cluster.status().
3. Router metadata_cache: add an option to fetch metadata/auth through the
   admin port of each member, separate from the routing destinations. Grant
   the router account SERVICE_CONNECTION_ADMIN when the option is used.
4. Until then, document that operator, router, recovery and backup accounts
   share max_connections with application traffic, how many connections they
   need, and that only one CONNECTION_ADMIN extra slot exists.

Portions of this website are copyright © 2001, 2002 The PHP Group

Page generated in 0.051 sec. using MySQL 8.4.7-u6-cloud

Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the respective copyright holders.
It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.