Bug #42055 ConcurrentModificationException in LoadBalancingConnectionProxy

• Submitted: 12 Jan 2009 16:39
• Modified: 14 Jan 2009 16:26
• Reporter: Rafael Chies
• Status: Closed
• Category: Connector / J
• Severity: S3 (Non-critical)
• Version: 5.1.7
• OS: Linux
• Assigned to: Todd Farmer
• CPU Architecture: Any

[12 Jan 2009 16:39] Rafael Chies

Description:
I've got an ConcurrentModificationException in LoadBalancingConnectionProxy:

java.util.ConcurrentModificationException
 at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
 at java.util.HashMap$KeyIterator.next(Unknown Source)
 at com.mysql.jdbc.LoadBalancingConnectionProxy.getGlobalBlacklist(LoadBalancingConnectionProxy.java:520)
 at com.mysql.jdbc.RandomBalanceStrategy.pickConnection(RandomBalanceStrategy.java:55)
 at com.mysql.jdbc.LoadBalancingConnectionProxy.pickNewConnection(LoadBalancingConnectionProxy.java:414)
 at com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:390)

How to repeat:
I did an unit test to be sure about the problem. But I had to comment the call to the pickNewConnection() method in the constructor. 

public class LoadBalancingConnectionProxyTest extends TestCase {

	LoadBalancingConnectionProxy proxy;

	public void setUp() throws Exception{
		Properties properties = new Properties();
		properties.put("loadBalanceBlacklistTimeout", "1000");
		List<String> hosts = new ArrayList<String>();
		hosts.add("host1");
		hosts.add("host2");
		hosts.add("host3");
		proxy = new LoadBalancingConnectionProxy(hosts, properties);
	}

	public void testGetGlobalBlacklist() throws Exception {
		proxy.addToGlobalBlacklist("host1");
		proxy.addToGlobalBlacklist("host2");
		Thread.sleep(1000); //Timeout set in setUp method. It's necessary to remove the host from blackList
		Map globalBlacklist = proxy.getGlobalBlacklist();
		System.out.println(globalBlacklist.size());
		assertEquals(0, globalBlacklist.size());
	}
}

Suggested fix:
I did a copy of the keys Set to be used in the loop. 

Set keysToAvoidConcurrentModification = new HashSet();
keysToAvoidConcurrentModification.addAll(keys);

for(Iterator i = keysToAvoidConcurrentModification.iterator(); i.hasNext(); ) {
     String host = (String) i.next();
    // OK if null is returned because another thread already purged Map entry.
    Long timeout = (Long) globalBlacklist.get(host);
    if(timeout != null && timeout.longValue() < System.currentTimeMillis()){
	// Timeout has expired, remove from blacklist
	synchronized(globalBlacklist){
        	globalBlacklist.remove(host);
	}
	blacklistClone.remove(host);
     }
}

[13 Jan 2009 23:47] Todd Farmer

Verified as described; problem is:

blacklistClone.remove(host);

while inside the Iterator loop on the keys from that Map.  Should instead use the Iterator.remove(); method:

i.remove();

[14 Jan 2009 0:08] Todd Farmer

Fixed in http://lists.mysql.com/commits/63188 .

[14 Jan 2009 16:26] Tony Bedford

An entry was added to the 5.1.8 changelog:

A ConcurrentModificationException was generated in LoadBalancingConnectionProxy:

java.util.ConcurrentModificationException
 at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
 at java.util.HashMap$KeyIterator.next(Unknown Source)
 at
com.mysql.jdbc.LoadBalancingConnectionProxy.getGlobalBlacklist(LoadBalancingConnectionProxy.java:520)
 at com.mysql.jdbc.RandomBalanceStrategy.pickConnection(RandomBalanceStrategy.java:55)
 at
com.mysql.jdbc.LoadBalancingConnectionProxy.pickNewConnection(LoadBalancingConnectionProxy.java:414)
 at
com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:390)