Description:
Originally reported at:
https://bugs.launchpad.net/bugs/1528260
mysql-proxy does not re-resolve DNS names of host in the event of connect failure.
If a backend mysql server's DNS name rotates it's IP address, the mysql-proxy will not correct it's lookup until the service is restarted.
The example below shows the service still attempting to connect to 10.0.0.2 days after the dns name in the launch config ip was moved to 10.0.0.3.
/usr/bin/mysql-proxy --plugins=proxy --log-level=debug --proxy-address=10.0.0.5:3306 --proxy-backend-addresses=mysql.proxy.local:3306 --daemon --log-use-syslog --pid-file=/var/run/mysql-proxy/mysql-proxy.pid
<snip...>
Dec 14 06:58:59 localhost mysql-proxy: 2015-12-14 06:58:59: (message) proxy-plugin.c.1503: connect(10.0.0.2:3306) failed: Connection timed out. Retrying with different backend.
Dec 14 06:58:59 localhost mysql-proxy: 2015-12-14 06:58:59: (critical) proxy-plugin.c.1612: Cannot connect, all backends are down.
Dec 14 06:58:59 localhost mysql-proxy: 2015-12-14 06:58:59: (debug) [network-mysqld.c:937]: error on a connection (fd: -1 event: 0). closing client connection.
Dec 14 06:58:59 localhost mysql-proxy: 2015-12-14 06:58:59: (warning) (libevent) event_del: event has no event_base set.
<snip>
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: mysql-proxy 0.8.1-1.1build1
ProcVersionSignature: Ubuntu 3.13.0-46.77-generic 3.13.11-ckt15
Uname: Linux 3.13.0-46-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.18
Architecture: amd64
Date: Mon Dec 21 15:23:22 2015
Ec2AMI: ami-018c9568
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1c
Ec2InstanceType: m3.large
Ec2Kernel: aki-919dcaf8
Ec2Ramdisk: unavailable
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: mysql-proxy
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.default.mysql.proxy: [modified]
modified.conffile..etc.init.d.mysql.proxy: [deleted]
mtime.conffile..etc.default.mysql.proxy: 2014-10-01T03:22:33.420509
How to repeat:
Create a mysql-proxy connnection to a backend server with a DNS name as the backend=address. (example: mysql.proxy.local)
Use a client to connect to the proxy. Success.
Change the ip address of the DNS entry for the backend-address server.
Flush DNS cache.
Use a client to connect to the proxy. Failure, no backends will be available.
Suggested fix:
On error "connect(x.x.x.x:3306) failed: Connection timed out. Retrying with different backend."
The host backedn 'mysql.proxy.local' which resolved to x.x.x.x should be re-resolved so that y.y.y.y can be used immediately upon connection failure.
