Description:
Hello,
We are developing a Java Web Spring and Hibernate-based application, connecting to a MySQL cluster having one management node, 2 NDB nodes and 2 SQL API Nodes deployed as below:
1) The management node is deployed on a virtual machine running Red Hat Enterprise Linux Server release 6.6 (Santiago) OS with 4GBs of RAM and 1 CPU;
2) We have 2 other virtual machines, with the same OS and the same resources as the one where the management node is deployed. On each of the 2 machines 1 NDB node and 1 SQL API Node are deployed (We will attach the configuration files afterwards).
3) From our application we query the database and after multiple reloading of certain pages we loose the connection to the database, because one of the 2 SQL API Nodes gets restarted. In our application server logs we get the following exception:
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 33 milliseconds ago. The last packet sent successfully to the server was 33 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1036)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3427)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3327)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2030)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:83)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:82)
at org.hibernate.loader.Loader.getResultSet(Loader.java:2066)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1863)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1839)
at org.hibernate.loader.Loader.doQuery(Loader.java:910)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:355)
at org.hibernate.loader.Loader.doList(Loader.java:2554)
at org.hibernate.loader.Loader.doList(Loader.java:2540)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2370)
at org.hibernate.loader.Loader.list(Loader.java:2365)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:497)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:387)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:236)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1264)
at org.hibernate.internal.QueryImpl.list(QueryImpl.java:103)
at wso2.api.wrapper.dao.impl.HibernateSubscriptionDaoImpl.getAPISubscriptionRequest(HibernateSubscriptionDaoImpl.java:183)
at wso2.api.wrapper.bo.subscription.impl.SubscriptionBoImpl.getAPISubscriptionRequestHistory(SubscriptionBoImpl.java:307)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
... 80 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2914)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3337)
... 115 more
We have associated this exception with the moment when the SQL API Node restarts . In the node's error logs we can see the following errors:
/pb2/build/sb_0-14878975-1427908940.3/rpm/BUILD/mysql-cluster-gpl-7.4.6/mysql-cluster-gpl-7.4.6/storage/ndb/src/common/util/NdbSqlUtil.cpp:480: require((lb + m1 <= n1 && lb + m2 <= n2)) failed
11:50:12 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=90
max_threads=151
thread_count=52
connection_count=51
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68104 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x3529a40
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
...............................
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
160616 13:50:12 mysqld_safe Number of processes running now: 0
160616 13:50:12 mysqld_safe mysqld restarted
2016-06-16 13:50:14 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-06-16 13:50:14 0 [Note] /usr/sbin/mysqld (mysqld 5.6.24-ndb-7.4.6-cluster-gpl) starting as process 28071 ...
2016-06-16 13:50:14 28071 [Note] Plugin 'FEDERATED' is disabled.
How to repeat:
We have a page that generates multiple database queries. After multiple reloads the page does not load anymore. In our application server logs (Tomcat8) we can see the "Communications link failure" exception at that moment in time. Navigating to both of the node's error logs, we can see that the SQL API corresponding to one node has restarted at a time stamp before the time stamp of the exception received in the application server logs.
