Bug #121670 Operator/Router can't connect when clients exhaust max_connections
Description:
MySQL Operator and the MySQL Router it deploys reach every InnoDB Cluster member
only through the regular client port (3306). They share max_connections with all
application traffic. When client connections fill max_connections, the operator
and the routers can no longer open sessions. Automatic recovery stops, and after
a primary failure writes do not resume until clients release connections.
Details:
- The operator hard-codes the member port: MySQLPod.port = 3306
(mysqloperator/controller/innodbcluster/cluster_api.py:3042). The
InnoDBCluster CR has no field to use the administrative interface
(admin_address/admin_port), although MySQL provides that interface for this
exact situation.
- The server allows only ONE extra connection beyond max_connections for
CONNECTION_ADMIN accounts. All of these accounts share it: localroot (sidecar),
root, mysqladmin-* (operator), mysql_innodb_cluster_* (recovery). The router
account (mysqlrouter-*) and mysqlbackup do not have CONNECTION_ADMIN.
- In steady state the internal accounts hold about 12 connections per member
(operator 6, router 2, recovery 1-2, event_scheduler 1). A single
rejoin/diagnose cycle opens several more.
Observed (max_connections=40, application account without admin privileges,
connections held through the router):
- Secondary mysqld killed: GR continued with 2 members. The operator failed to
connect to the remaining members with "1040 Too many connections", left the
instance in diagnose state OFFLINE_UNCERTAIN and retried every 60 s. No rejoin
for 8+ minutes. After the client connections were released, the instance
rejoined 70 s later.
- Primary mysqld killed: GR elected a new primary. However, all routers got 1040
from all 3 members and logged "Failed fetching metadata from any of the 3
metadata servers". The RW port did not follow the new primary, and writes were
impossible for 8+ minutes. Recovery came 72 s after the client connections were
released.
- Saturation alone, without a failure, did not show errors within 3 minutes. The
operator only opens SQL sessions when it reconciles, so the problem appears
when a failure coincides with a connection spike. Failovers themselves often
cause such spikes.
Workaround attempts that failed:
- Swapping ports (port=3336 for clients, admin_port=3306 + admin_address=<pod
FQDN>, so the operator's 3306 hits the admin interface): cluster creation
never finishes (InnoDBCluster stays INITIALIZING, 0 members). The operator
matches members by the string "<fqdn>:3306" (pod.endpoint) in
cluster.status(). AdminAPI records the members as "<fqdn>:@@port" (:3336), so
the operator logs "Could not find <pod> in local cluster.status() output"
forever. Setting report_port=3306 instead would make Router send application
traffic to the admin port, which is not usable either.
- The only remaining mitigation is to keep total client connections well below
max_connections (per-account MAX_USER_CONNECTIONS, router
max_total_connections). With 10 spare slots we still saw intermittent 1040 for
router/operator. With 20 spare slots recovery was normal (router failover 22 s,
rejoin about 1 min 50 s).
Related: mysql/mysql-operator GitHub issue #69 (router metadata refresh blocked
by 1040 on the new primary after failover). Bug #99917 (admin interface
connection limits).
How to repeat:
1. Deploy operator 26.7.0-2.3.0 and a 3-member InnoDBCluster (server 9.7.1,
router 26.7.1, 2 router instances).
2. On every member: SET PERSIST max_connections = 40;
3. Create an application account without CONNECTION_ADMIN /
SERVICE_CONNECTION_ADMIN:
CREATE USER 'cust'@'%' IDENTIFIED BY '...';
GRANT SELECT, INSERT, UPDATE, DELETE ON test.* TO 'cust'@'%';
4. From a client pod, open as many connections as possible as 'cust' through
the router: RW port 6446 (~60) and RO port 6447 (~120). Keep them idle and
reopen any dropped connection every second, so that every member stays at
max_connections.
5. Secondary case: kill -9 mysqld in a SECONDARY pod.
Watch the operator log (kubectl logs deploy/mysql-operator) and
performance_schema.replication_group_members.
-> operator: "1040 Too many connections", diagnose OFFLINE_UNCERTAIN, the
member does not rejoin while the client connections are held.
6. Release the client connections -> the member rejoins about 70 s later.
7. Primary case: repeat step 4, then kill -9 mysqld in the PRIMARY pod.
-> router log: "Failed fetching metadata from any of the 3 metadata
servers"; writes through 6446 keep failing; no rejoin.
8. Release the client connections -> service recovers about 72 s later.
Suggested fix:
1. Allow the operator's control sessions to use the administrative interface.
For example, add a CR option that enables admin_address=<pod FQDN/IP> and
admin_port per member. The operator then connects to that port.
mysqladmin-* already has SERVICE_CONNECTION_ADMIN via GRANT ALL.
2. Separate "member identity" (the metadata address <report_host>:<port>) from
the "operator connect port" in MySQLPod.endpoint / endpoint_co and in
diagnose.py. Changing the connect port must not break member matching
against cluster.status().
3. Router metadata_cache: add an option to fetch metadata/auth through the
admin port of each member, separate from the routing destinations. Grant
the router account SERVICE_CONNECTION_ADMIN when the option is used.
4. Until then, document that operator, router, recovery and backup accounts
share max_connections with application traffic, how many connections they
need, and that only one CONNECTION_ADMIN extra slot exists.
Portions of this website are copyright © 2001, 2002 The PHP Group
Page generated in 0.051 sec. using MySQL 8.4.7-u6-cloud
Timestamp references displayed by the system are UTC.
Content reproduced on this site is the property of the
respective copyright holders.
It is not reviewed in advance
by Oracle and does not necessarily represent the opinion of
Oracle or any other party.