Bug #58706 Connection failover leaving slave/secondary read-only.

• Submitted: 3 Dec 2010 16:37
• Modified: 10 Jan 2011 12:53
• Reporter: Matthew Montgomery
• Status: Closed
• Category: Connector / J
• Severity: S2 (Serious)
• Version: 5.1.13
• OS: Any
• CPU Architecture: Any

[3 Dec 2010 16:37] Matthew Montgomery

Description:
Failover attempts between two read-write masters did not properly set this.currentConn.setReadOnly(false);

The shouldFallBack() is always returning (true) due to a counter not being reset.

How to repeat:
Setup 2 node circular replication for active-passive failover.

jdbc:mysql://host1,host2/reporting?failOverReadOnly=false&autoReconnectForPools=true&loadBalanceAutoCommitStatementThreshold=100&cacheCallableStmts=true&cachePrepStmts=true&connectTimeout=2000&socketTimeout=2000

Suggested fix:
Contributed fix:

--- FailoverConnectionProxy.java	2010-06-23 10:38:34.000000000 -0500
+++ FailoverConnectionProxy.java	2010-12-03 10:19:29.863892600 -0600
@@ -122,6 +122,14 @@
 			createPrimaryConnection();
 			
 			return super.invoke(proxy, method, args);
+		} else if ("isClosed".equals(methodName)) {
+			InvocationHandler h=Proxy.getInvocationHandler(proxy);
+			if (h instanceof FailoverConnectionProxy) {
+				if (( ((FailoverConnectionProxy) h).failedOver) && shouldFallBack()) {
+				createPrimaryConnection();
+				return super.invoke(proxy, method, args);
+				}
+			}
 		}
 
 		return super.invoke(proxy, method, args);
@@ -179,7 +187,8 @@
 		}
 		
 		super.pickNewConnection();
-		this.currentConn.setReadOnly(true);
+		// this.currentConn.setReadOnly(true);
+		this.currentConn.setReadOnly(false);
 		this.failedOver = true;
 	}
 
@@ -193,9 +202,16 @@
 	private boolean shouldFallBack() {
 		long secondsSinceFailedOver = (System.currentTimeMillis() - this.masterFailTimeMillis) / 1000;
 
+		if (secondsSinceFailedOver >= this.secondsBeforeRetryMaster) {
+			this.masterFailTimeMillis=System.currentTimeMillis();
+			return true;
+		} else {
+			return false;
+		}
+
 		// Done this way so we can set a condition in the debugger
-		boolean tryFallback = ((secondsSinceFailedOver >= this.secondsBeforeRetryMaster) || (this.queriesIssuedFailedOver >= this.queriesBeforeRetryMaster));
+		// boolean tryFallback = ((secondsSinceFailedOver >= this.secondsBeforeRetryMaster) || (this.queriesIssuedFailedOver >= this.queriesBeforeRetryMaster));
 
-		return tryFallback;
+		// return tryFallback;
 	}
 }

[6 Dec 2010 16:35] Mark Matthews

Fixed for 5.1.13.

[6 Dec 2010 16:35] Mark Matthews

Sorry, 5.1.14.

[10 Jan 2011 12:53] Tony Bedford

An entry has been added to the 5.1.14 changelog:

Connection failover left slave/secondary in read-only mode. Failover attempts between two read-write masters did not properly set this.currentConn.setReadOnly(false).