Bug #78741 ConcurrentModificationException exception cause application to crash
Submitted: 7 Oct 2015 18:36 Modified: 22 Apr 2016 1:33
Reporter: Imran Majeed Email Updates:
Status: No Feedback Impact on me:
None 
Category:Connector / J Severity:S2 (Serious)
Version:5.1.36 OS:Windows (Window 7 and Window server 2012)
Assigned to: Filipe Silva CPU Architecture:Any
Tags: fabric

[7 Oct 2015 18:36] Imran Majeed
Description:
Application crashes when trying to add slaves at startup.

Same thread is iterating and modifying the map, synchronization does not solve the problem in this case. Use a map which allows modification during iteration.

How to repeat:
1= configure and fabric node with one master and one slave. All are using 5.6.26
2= create empty database as liferay
2= download liferay (community or eterprise 6.2GA4) and configure portal-ext.properties
as following
jdbc.default.driverClassName=com.mysql.fabric.jdbc.FabricMySQLDriver
jdbc.default.username=liferay
jdbc.default.password=password
db.name=liferay
db.host=localhost
db.port=32274
jdbc.default.url=jdbc:mysql:fabric://${db.host}:${db.port}/${db.name}?useUnicode=yes&characterEncoding=UTF-8&useFastDateParsing=false&fabricServerGroup=liferay_group&fabricUsername=admin&fabricPassword=admin&mode=fabric.MODE_READWRITE&fabricReportErrors=true

3= it will crash right after creating all tables with following error

17:59:02,837 INFO  [localhost-startStop-1][DialectDetector:71] Determine dialect for MySQL -1
17:59:02,862 INFO  [localhost-startStop-1][DialectDetector:136] Found dialect org.hibernate.dialect.MySQLDialect
Starting Liferay Portal Enterprise Edition 6.2.10 EE GA1 (Newton / Build 6210 / November 1, 2013)
17:59:18,377 INFO  [localhost-startStop-1][StartupAction:99] The following patches are installed: portal-63-6210
17:59:19,302 INFO  [localhost-startStop-1][BaseDB:484] Database does not support case sensitive queries
17:59:19,582 ERROR [localhost-startStop-1][JDBCExceptionReporter:82] An SQLException was provoked by the following failure: java.util.ConcurrentModificationException
17:59:19,587 ERROR [localhost-startStop-1][CustomSQLUtil:142] java.lang.RuntimeException: Unable to get class name from value com.liferay.portlet.messageboards.model.MBThread
java.lang.RuntimeException: Unable to get class name from value com.liferay.portlet.messageboards.model.MBThread
	at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.getClassNameId(ClassNameLocalServiceImpl.java:157)
	at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.getClassNameId(ClassNameLocalServiceImpl.java:145)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:115)
	at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:175)
	at com.sun.proxy.$Proxy21.getClassNameId(Unknown Source)
	at com.liferay.portal.service.ClassNameLocalServiceUtil.getClassNameId(ClassNameLocalServiceUtil.java:306)
	at com.liferay.portal.util.PortalImpl.getClassNameId(PortalImpl.java:1423)
	at com.liferay.portal.util.PortalImpl.initCustomSQL(PortalImpl.java:5781)
	at com.liferay.portal.util.PortalUtil.initCustomSQL(PortalUtil.java:1645)
	at com.liferay.util.dao.orm.CustomSQL.reloadCustomSQL(CustomSQL.java:317)
	at com.liferay.util.dao.orm.CustomSQL.<init>(CustomSQL.java:84)
	at com.liferay.util.dao.orm.CustomSQLUtil.<init>(CustomSQLUtil.java:139)
	at com.liferay.util.dao.orm.CustomSQLUtil.<clinit>(CustomSQLUtil.java:148)
	at com.liferay.portal.tools.DBUpgrader.upgrade(DBUpgrader.java:126)
	at com.liferay.portal.events.StartupAction.doRun(StartupAction.java:180)
	at com.liferay.portal.ee.license.StartupAction.doRun(Unknown Source)
	at com.liferay.portal.events.StartupAction.run(StartupAction.java:73)
	at com.liferay.portal.servlet.MainServlet.processStartupEvents(MainServlet.java:1247)
	at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:209)
	at javax.servlet.GenericServlet.init(GenericServlet.java:160)
	at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1280)
	at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1193)
	at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1088)
	at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5176)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5460)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:656)
	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1635)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.GenericJDBCException: could not execute query
	at com.liferay.portal.service.persistence.impl.BasePersistenceImpl.processException(BasePersistenceImpl.java:251)
	at com.liferay.portal.service.persistence.ClassNamePersistenceImpl.fetchByValue(ClassNamePersistenceImpl.java:224)
	at com.liferay.portal.service.persistence.ClassNamePersistenceImpl.fetchByValue(ClassNamePersistenceImpl.java:135)
	at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.addClassName(ClassNameLocalServiceImpl.java:41)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:115)
	at com.liferay.portal.spring.transaction.DefaultTransactionExecutor.execute(DefaultTransactionExecutor.java:62)
	at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:51)
	at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
	at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:175)
	at com.sun.proxy.$Proxy21.addClassName(Unknown Source)
	at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.getClassName(ClassNameLocalServiceImpl.java:134)
	at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.getClassNameId(ClassNameLocalServiceImpl.java:152)
	... 39 more
Caused by: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.GenericJDBCException: could not execute query
	at com.liferay.portal.dao.orm.hibernate.ExceptionTranslator.translate(ExceptionTranslator.java:30)
	at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:125)
	at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:98)
	at com.liferay.portal.service.persistence.ClassNamePersistenceImpl.fetchByValue(ClassNamePersistenceImpl.java:200)
	... 53 more
Caused by: org.hibernate.exception.GenericJDBCException: could not execute query
	at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
	at org.hibernate.loader.Loader.doList(Loader.java:2545)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
	at org.hibernate.loader.Loader.list(Loader.java:2271)
	at org.hibernate.hql.classic.QueryTranslatorImpl.list(QueryTranslatorImpl.java:940)
	at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
	at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268)
	at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
	at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:113)
	... 55 more
Caused by: java.sql.SQLException: An SQLException was provoked by the following failure: java.util.ConcurrentModificationException
	at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
	at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
	at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:74)
	at com.mchange.v2.c3p0.impl.NewPooledConnection.handleThrowable(NewPooledConnection.java:492)
	at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:179)
	at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
	at com.sun.proxy.$Proxy4.prepareStatement(Unknown Source)
	at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
	at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
	at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
	at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1700)
	at org.hibernate.loader.Loader.doQuery(Loader.java:801)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
	at org.hibernate.loader.Loader.doList(Loader.java:2542)
	... 62 more
Caused by: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
	at java.util.HashMap$ValueIterator.next(HashMap.java:950)
	at com.mysql.jdbc.ReplicationConnectionGroup.addSlaveHost(ReplicationConnectionGroup.java:98)
	at com.mysql.fabric.jdbc.FabricMySQLConnectionProxy.syncGroupServersToReplicationConnectionGroup(FabricMySQLConnectionProxy.java:500)
	at com.mysql.fabric.jdbc.FabricMySQLConnectionProxy.getActiveConnection(FabricMySQLConnectionProxy.java:550)
	at com.mysql.fabric.jdbc.FabricMySQLConnectionProxy.prepareStatement(FabricMySQLConnectionProxy.java:735)
	at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:162)
	... 74 more
17:59:19,593 ERROR [localhost-startStop-1][MainServlet:212] java.lang.NullPointerException
java.lang.NullPointerException
	at com.liferay.util.dao.orm.CustomSQLUtil.reloadCustomSQL(CustomSQLUtil.java:90)
	at com.liferay.portal.tools.DBUpgrader.upgrade(DBUpgrader.java:126)
	at com.liferay.portal.events.StartupAction.doRun(StartupAction.java:180)
	at com.liferay.portal.ee.license.StartupAction.doRun(Unknown Source)
	at com.liferay.portal.events.StartupAction.run(StartupAction.java:73)
	at com.liferay.portal.servlet.MainServlet.processStartupEvents(MainServlet.java:1247)
	at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:209)
	at javax.servlet.GenericServlet.init(GenericServlet.java:160)
	at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1280)
	at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1193)
	at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1088)
	at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5176)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5460)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:656)
	at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1635)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

Suggested fix:
Option A
if map named as "replicationConnections" in "ReplicationConnectionGroup.java" is allowed to have concurrent operation then change declaration like this

private Map<Long, ReplicationConnection> replicationConnections = new ConcurrentHashMap<Long, ReplicationConnection>();

because following line of code .. iterating and changing the map in same loop

 for (ReplicationConnection c : this.replicationConnections.values()) {
            c.addSlaveHost(host);
        } 

Option B

Catch Exception under SQL Exception in following line of code taken form "FabricMySQLConnectionProxy.java" in method syncGroupServersToReplicationConnectionGroup(). Changes are start and end with comments.

// 2. remove any old slaves from the connection group
            for (String hostPortString : replConnGroup.getSlaveHosts()) {
                Server fabServer = this.serverGroup.getServer(hostPortString);
                if (fabServer == null || !(fabServer.isSlave())) {
                    try {
                        replConnGroup.removeSlaveHost(hostPortString, true);
                    } catch (SQLException ex) {
                        // effectively ignored
                        getLog().logWarn("Unable to remove slave: " + hostPortString, ex);
                    }
/******start of change *************/
catch (Exception e){
                    getLog().logWarn("Unable to add slave: java.util.ConcurrentModificationException " + addSlaveHost, e);
                }

/**********end of change ***********/
                }
            }
[7 Oct 2015 18:49] Imran Majeed
Option B ( loop shows in previous post was wrong. This is correct)

Catch Exception under SQL Exception in following line of code taken form "FabricMySQLConnectionProxy.java" in method syncGroupServersToReplicationConnectionGroup(). Changes are start and end with comments.

 // 1. add any new slaves to the connection group
            for (Server s : this.serverGroup.getServers()) {
                if (s.isSlave()) {
                    // this is a no-op if the slave is already present
                    try {
                        replConnGroup.addSlaveHost(s.getHostPortString());
                    } catch (SQLException ex) {
                        // effectively ignored
                        getLog().logWarn("Unable to add slave: " + s.toString(), ex);
                    }
/******start of change *************/
catch (Exception e){
                    getLog().logWarn("Unable to add slave: java.util.ConcurrentModificationException " + addSlaveHost, e);
                }

/**********end of change ***********/
                }
            }
[22 Mar 2016 1:33] Filipe Silva
Hi Imran,

This bug should be already fixed in Connector/J 5.1.38. Can you try it out please?

Thank you,
[23 Apr 2016 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".