Description:
Application crashes when trying to add slaves at startup.
Same thread is iterating and modifying the map, synchronization does not solve the problem in this case. Use a map which allows modification during iteration.
How to repeat:
1= configure and fabric node with one master and one slave. All are using 5.6.26
2= create empty database as liferay
2= download liferay (community or eterprise 6.2GA4) and configure portal-ext.properties
as following
jdbc.default.driverClassName=com.mysql.fabric.jdbc.FabricMySQLDriver
jdbc.default.username=liferay
jdbc.default.password=password
db.name=liferay
db.host=localhost
db.port=32274
jdbc.default.url=jdbc:mysql:fabric://${db.host}:${db.port}/${db.name}?useUnicode=yes&characterEncoding=UTF-8&useFastDateParsing=false&fabricServerGroup=liferay_group&fabricUsername=admin&fabricPassword=admin&mode=fabric.MODE_READWRITE&fabricReportErrors=true
3= it will crash right after creating all tables with following error
17:59:02,837 INFO [localhost-startStop-1][DialectDetector:71] Determine dialect for MySQL -1
17:59:02,862 INFO [localhost-startStop-1][DialectDetector:136] Found dialect org.hibernate.dialect.MySQLDialect
Starting Liferay Portal Enterprise Edition 6.2.10 EE GA1 (Newton / Build 6210 / November 1, 2013)
17:59:18,377 INFO [localhost-startStop-1][StartupAction:99] The following patches are installed: portal-63-6210
17:59:19,302 INFO [localhost-startStop-1][BaseDB:484] Database does not support case sensitive queries
17:59:19,582 ERROR [localhost-startStop-1][JDBCExceptionReporter:82] An SQLException was provoked by the following failure: java.util.ConcurrentModificationException
17:59:19,587 ERROR [localhost-startStop-1][CustomSQLUtil:142] java.lang.RuntimeException: Unable to get class name from value com.liferay.portlet.messageboards.model.MBThread
java.lang.RuntimeException: Unable to get class name from value com.liferay.portlet.messageboards.model.MBThread
at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.getClassNameId(ClassNameLocalServiceImpl.java:157)
at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.getClassNameId(ClassNameLocalServiceImpl.java:145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:115)
at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:175)
at com.sun.proxy.$Proxy21.getClassNameId(Unknown Source)
at com.liferay.portal.service.ClassNameLocalServiceUtil.getClassNameId(ClassNameLocalServiceUtil.java:306)
at com.liferay.portal.util.PortalImpl.getClassNameId(PortalImpl.java:1423)
at com.liferay.portal.util.PortalImpl.initCustomSQL(PortalImpl.java:5781)
at com.liferay.portal.util.PortalUtil.initCustomSQL(PortalUtil.java:1645)
at com.liferay.util.dao.orm.CustomSQL.reloadCustomSQL(CustomSQL.java:317)
at com.liferay.util.dao.orm.CustomSQL.<init>(CustomSQL.java:84)
at com.liferay.util.dao.orm.CustomSQLUtil.<init>(CustomSQLUtil.java:139)
at com.liferay.util.dao.orm.CustomSQLUtil.<clinit>(CustomSQLUtil.java:148)
at com.liferay.portal.tools.DBUpgrader.upgrade(DBUpgrader.java:126)
at com.liferay.portal.events.StartupAction.doRun(StartupAction.java:180)
at com.liferay.portal.ee.license.StartupAction.doRun(Unknown Source)
at com.liferay.portal.events.StartupAction.run(StartupAction.java:73)
at com.liferay.portal.servlet.MainServlet.processStartupEvents(MainServlet.java:1247)
at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:209)
at javax.servlet.GenericServlet.init(GenericServlet.java:160)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1280)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1193)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1088)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5176)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5460)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:656)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1635)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.liferay.portal.kernel.exception.SystemException: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.GenericJDBCException: could not execute query
at com.liferay.portal.service.persistence.impl.BasePersistenceImpl.processException(BasePersistenceImpl.java:251)
at com.liferay.portal.service.persistence.ClassNamePersistenceImpl.fetchByValue(ClassNamePersistenceImpl.java:224)
at com.liferay.portal.service.persistence.ClassNamePersistenceImpl.fetchByValue(ClassNamePersistenceImpl.java:135)
at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.addClassName(ClassNameLocalServiceImpl.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:115)
at com.liferay.portal.spring.transaction.DefaultTransactionExecutor.execute(DefaultTransactionExecutor.java:62)
at com.liferay.portal.spring.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:51)
at com.liferay.portal.spring.aop.ServiceBeanMethodInvocation.proceed(ServiceBeanMethodInvocation.java:111)
at com.liferay.portal.spring.aop.ServiceBeanAopProxy.invoke(ServiceBeanAopProxy.java:175)
at com.sun.proxy.$Proxy21.addClassName(Unknown Source)
at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.getClassName(ClassNameLocalServiceImpl.java:134)
at com.liferay.portal.service.impl.ClassNameLocalServiceImpl.getClassNameId(ClassNameLocalServiceImpl.java:152)
... 39 more
Caused by: com.liferay.portal.kernel.dao.orm.ORMException: org.hibernate.exception.GenericJDBCException: could not execute query
at com.liferay.portal.dao.orm.hibernate.ExceptionTranslator.translate(ExceptionTranslator.java:30)
at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:125)
at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:98)
at com.liferay.portal.service.persistence.ClassNamePersistenceImpl.fetchByValue(ClassNamePersistenceImpl.java:200)
... 53 more
Caused by: org.hibernate.exception.GenericJDBCException: could not execute query
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:140)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:128)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2545)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
at org.hibernate.loader.Loader.list(Loader.java:2271)
at org.hibernate.hql.classic.QueryTranslatorImpl.list(QueryTranslatorImpl.java:940)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
at com.liferay.portal.dao.orm.hibernate.QueryImpl.list(QueryImpl.java:113)
... 55 more
Caused by: java.sql.SQLException: An SQLException was provoked by the following failure: java.util.ConcurrentModificationException
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:74)
at com.mchange.v2.c3p0.impl.NewPooledConnection.handleThrowable(NewPooledConnection.java:492)
at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:179)
at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376)
at com.sun.proxy.$Proxy4.prepareStatement(Unknown Source)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:534)
at org.hibernate.jdbc.AbstractBatcher.getPreparedStatement(AbstractBatcher.java:452)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1700)
at org.hibernate.loader.Loader.doQuery(Loader.java:801)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
at org.hibernate.loader.Loader.doList(Loader.java:2542)
... 62 more
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
at java.util.HashMap$ValueIterator.next(HashMap.java:950)
at com.mysql.jdbc.ReplicationConnectionGroup.addSlaveHost(ReplicationConnectionGroup.java:98)
at com.mysql.fabric.jdbc.FabricMySQLConnectionProxy.syncGroupServersToReplicationConnectionGroup(FabricMySQLConnectionProxy.java:500)
at com.mysql.fabric.jdbc.FabricMySQLConnectionProxy.getActiveConnection(FabricMySQLConnectionProxy.java:550)
at com.mysql.fabric.jdbc.FabricMySQLConnectionProxy.prepareStatement(FabricMySQLConnectionProxy.java:735)
at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:162)
... 74 more
17:59:19,593 ERROR [localhost-startStop-1][MainServlet:212] java.lang.NullPointerException
java.lang.NullPointerException
at com.liferay.util.dao.orm.CustomSQLUtil.reloadCustomSQL(CustomSQLUtil.java:90)
at com.liferay.portal.tools.DBUpgrader.upgrade(DBUpgrader.java:126)
at com.liferay.portal.events.StartupAction.doRun(StartupAction.java:180)
at com.liferay.portal.ee.license.StartupAction.doRun(Unknown Source)
at com.liferay.portal.events.StartupAction.run(StartupAction.java:73)
at com.liferay.portal.servlet.MainServlet.processStartupEvents(MainServlet.java:1247)
at com.liferay.portal.servlet.MainServlet.init(MainServlet.java:209)
at javax.servlet.GenericServlet.init(GenericServlet.java:160)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1280)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1193)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1088)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5176)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5460)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:656)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1635)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Suggested fix:
Option A
if map named as "replicationConnections" in "ReplicationConnectionGroup.java" is allowed to have concurrent operation then change declaration like this
private Map<Long, ReplicationConnection> replicationConnections = new ConcurrentHashMap<Long, ReplicationConnection>();
because following line of code .. iterating and changing the map in same loop
for (ReplicationConnection c : this.replicationConnections.values()) {
c.addSlaveHost(host);
}
Option B
Catch Exception under SQL Exception in following line of code taken form "FabricMySQLConnectionProxy.java" in method syncGroupServersToReplicationConnectionGroup(). Changes are start and end with comments.
// 2. remove any old slaves from the connection group
for (String hostPortString : replConnGroup.getSlaveHosts()) {
Server fabServer = this.serverGroup.getServer(hostPortString);
if (fabServer == null || !(fabServer.isSlave())) {
try {
replConnGroup.removeSlaveHost(hostPortString, true);
} catch (SQLException ex) {
// effectively ignored
getLog().logWarn("Unable to remove slave: " + hostPortString, ex);
}
/******start of change *************/
catch (Exception e){
getLog().logWarn("Unable to add slave: java.util.ConcurrentModificationException " + addSlaveHost, e);
}
/**********end of change ***********/
}
}