MySQL Bugs: #54619: Session factory in a different thread than sessions crashes JVM

Bug #54619	Session factory in a different thread than sessions crashes JVM
Submitted:	18 Jun 2010 18:04	Modified:	18 May 2018 15:11
Reporter:	Heikki Laaksonen	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	MySQL Cluster: Cluster/J	Severity:	S3 (Non-critical)
Version:	7.1.3	OS:	Any
Assigned to:	Assigned Account	CPU Architecture:	Any
Tags:	ClusterJ, crash, JVM, thread

Description:
If SessionFactory is initialized in a different thread than where sessions are created, session.newInstance call will crash JVM. This happens on Windows and Linux. In Windows the crash seems to be coming from DomainFieldHandlerImpl and from line table.getColumn(columnName);.

Workaround is to insert a dummy row (most likely any db access works) to database from same thread where the SessionFactory is created. After this the sessions seem to be working from other thread.

How to repeat:
package clusterj;

// Main.java

/*
 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0;
 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0;
 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='TRADITIONAL';

 CREATE SCHEMA IF NOT EXISTS `clusterj` DEFAULT CHARACTER SET latin1 COLLATE latin1_bin ;

 -- -----------------------------------------------------
 -- Table `clusterj`.`example`
 -- -----------------------------------------------------
 CREATE  TABLE IF NOT EXISTS `clusterj`.`example` (
 `number` BIGINT UNSIGNED NOT NULL ,
 PRIMARY KEY (`number`) ,
 UNIQUE INDEX `number_UNIQUE` USING HASH (`number` ASC) )
 ENGINE = ndbcluster;

 SET SQL_MODE=@OLD_SQL_MODE;
 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS;
 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS;
 */

// Set java.library.path variable to point to the directory containing the Cluster ndbclient library.
// -Djava.library.path=C:\Work\Programming\MySQL\MySQL-Cluster\mysql-cluster-gpl-7.1.3-win32\lib

import java.util.Properties;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import com.mysql.clusterj.ClusterJHelper;
import com.mysql.clusterj.Session;
import com.mysql.clusterj.SessionFactory;

public class Main {

	public static void main(String[] args) {

		/*
		 * MySQL Cluster generic Linux and Windows 7.1.3
		 * 
		 * Session factory in a different thread than sessions crashes JVM.
		 * 
		 * If SessionFactory is initialized in a different thread than where
		 * sessions are created, session.newInstance call will crash JVM. This
		 * happens on Windows and Linux. In Windows the crash seems to be coming
		 * from DomainFieldHandlerImpl and from line table.getColumn(columnName);.
		 * 
		 * Workaround is to insert a dummy row (most likely any db access works)
		 * from same thread where the SessionFactory is created. After this the
		 * session seem to be working from other thread.
		 */
		ExecutorService pool = Executors.newFixedThreadPool(2);
		// 1. Manager created in main thread,
		Manager manager = new Manager();
		
		//insert(1); // 2. Remove the comment to get this work.

		// 3. Inserting a row from different thread.
		pool.submit(new Insert(2));
	}

	private static void insert(long number) {
		Session session = Manager.getSessionFactory().getSession();
		IEntity entity = session.newInstance(IEntity.class);
		entity.setNumber(number);
		session.currentTransaction().begin();
		session.persist(entity);
		session.currentTransaction().commit();

		return;
	}
}

class Manager {

	static SessionFactory sessionfactory;

	Manager() {
		createClusterJSession();
		return;
	}

	private void createClusterJSession() {
		Properties properties = new Properties();
		properties.put("com.mysql.clusterj.connectstring", "localhost:1186");
		properties.put("com.mysql.clusterj.database", "clusterj");
		sessionfactory = ClusterJHelper.getSessionFactory(properties);

		return;
	}

	public static SessionFactory getSessionFactory() {
		return sessionfactory;
	}
}

class Insert implements Runnable {

	SessionFactory sessionfactory;
	long number;

	Insert(long number) {
		this.number = number;
		return;
	}

	private void insert(long number) {
		Session session = Manager.getSessionFactory().getSession();
		IEntity entity = session.newInstance(IEntity.class); // Crash from here.
		entity.setNumber(number);
		session.currentTransaction().begin();
		session.persist(entity);
		session.currentTransaction().commit();
		return;
	}

	@Override
	public void run() {
		insert(this.number);
	}
}

package clusterj;

// IEntity.java

import com.mysql.clusterj.annotation.PersistenceCapable;
import com.mysql.clusterj.annotation.PrimaryKey;

@PersistenceCapable(table = "example")
public interface IEntity {

	@PrimaryKey
	long getNumber();

	void setNumber(long number);
}

Main.java

Attachment: Main.java (application/octet-stream, text), 3.57 KiB.

IEntity.java

Attachment: IEntity.java (application/octet-stream, text), 290 bytes.

There were issues in 7.1.3 in the schema handling, and schema handling was rewritten in 7.1.4.

I'm unable to duplicate the problem in the current build. Can you try with 7.1.4 and see if it still occurs?

Craig, I have the same problem with version 7.1.4b while running both under Windows and Linux! I have 20 threads, and want to have session in each of them, while it is logically obvious that factory should exist in application only in one instance. So i create Session Factory in main thread and then while other threads are starting they try to get sessions from factory but JVM crashes.
Error occur inside C library ndbclient(both .dll and .so).
I fixed a bug by using own factory in each thread
As long as bug is not fixed please answer is it normal to have 20 factories in diffrend thread or it may cause some errors?

The use case is supported and should not crash. The expected usage is to have a single SessionFactory for multiple sessions in the same VM, and a single thread for each Session obtained from the SessionFactory.

Rewritten test case to use JUnit

Attachment: Bug54619.java (application/octet-stream, text), 3.48 KiB.

Could you try the JUnit test case to see if this test case stimulates the error? I'm unable to replicate in the latest code base. I'd like to know if the changes I made to your test affect the error.

Sorry, I have been busy with other tasks but here is a quick update.

The same problem exists also in 7.1.4b for me.

But the new information is that the problem exist only if the the session factory is created in the first "main" thread. If a new thread is started from main for session factory and session, it seems to work in Windows. I have had not yet time to check this in Linux.

I suspect this is a synchronization problem that you see on your 8-cpu machine but we have not yet been able to reproduce on smaller developer machines.  I have submitted a patch at http://lists.mysql.com/commits/112692 that may solve the problem.

I can reproduce this on Mac and Linux (32/64 bit) and dual cores. Invalid memory access happens in mysys/charset.c when my_read_charset_file() calls my_stat in pthread_once() init.

I can however not reproduce this with a simple multithreaded program from C simply constructing CharsetMap.

And in clusterj revno 321 it seems solved.

Still a problem...

This is still a problem in Connector/J 5.1.16. I can reliably reproduce it in a JBoss 6 J2EE application running on Windows Server 2008 x64. 

The use case is loosely as follows:

1) A SessionFactory is created during J2EE application startup by the application server's deployment process and the reference saved.

2) An arbitrary time later, another thread creates a Session from the SessionFactory, and calls the getInstance() method on it.

3) A Windows ACCESS_VIOLATION_EXCEPTION occurs in ndbclient.dll. The JVM catches the exception, produces an hs_err report, and terminates. 

I will attach the report. This was observed on a 2 CPU system, but I don't believe that is relevant.

JVM crash dump

Attachment: hs_err_pid3584.log (application/octet-stream, text), 29.16 KiB.

Is there any solution to this problem? Here is another simple test case that crashes jvm 1.6x,1.7x and 1.8x 

Note before i run the test i insert a single record 
insert into test values (1,"asd");

A simple test entity:

@PersistenceCapable(table = "test")
public interface TestEntity {

    @PrimaryKey
    @Column(name = "id")
    public int getId();

    public void setId(int id);

    @Column(name = "data")
    public String getData();

    public void setData(String data);

}

And the test case: 

 @Test
    public static void test() throws Exception {

	final Properties props = new Properties();

	final String host = "172.16.250.141";
	final String dbName = "core_db";
	props.put("com.mysql.clusterj.connectstring", host + ":1186");
	props.put("com.mysql.clusterj.database", dbName);

	final SessionFactory sessionFactory = ClusterJHelper.getSessionFactory(props);

	final Session first = sessionFactory.getSession();

	final Session second = sessionFactory.getSession();

	new Thread(new Runnable() {

	    @Override
	    public void run() {
		Session session = first;
		try {
		    session.currentTransaction().begin();
		    final TestEntity testEntity = session.newInstance(TestEntity.class);

		    testEntity.setId(1);
		    testEntity.setData("data");

		    session.persist(testEntity);

		    session.currentTransaction().commit();

		} catch (Exception e) {
		    e.printStackTrace();
		} finally {
		    try {
			if (session != null) {
			    session.currentTransaction().rollback();
			    session.close();
			}
		    } catch (Exception e) {
			e.printStackTrace();
		    }
		}

		try {
		    sessionFactory.close();
		} catch (Exception e) {
		    e.printStackTrace();

		}

	    }
	}).start();

	Thread.currentThread().sleep(1000L);

	final SessionFactory newSessionFactory = ClusterJHelper.getSessionFactory(props);

	Thread.currentThread().sleep(1000L);

	System.out.println("Second try...");

	new Thread(new Runnable() {

	    @Override
	    public void run() {
		Session session = second;
		try {
		    session.currentTransaction().begin();

		    System.err.println("Before new instance");

		    final TestEntity testEntity = session.newInstance(TestEntity.class);

		    testEntity.setId(1);
		    testEntity.setData("data");

		    session.persist(testEntity);

		    session.currentTransaction().commit();

		} catch (Exception e) {
		    e.printStackTrace();
		} finally {
		    try {
			if (session != null) {
			    session.currentTransaction().rollback();
			    session.close();
			}
		    } catch (Exception e) {
			e.printStackTrace();
		    }
		}

		try {
		    sessionFactory.close();
		} catch (Exception e) {
		    e.printStackTrace();
		}

	    }
	}).start();

	while (true) {
	    Thread.currentThread().sleep(10000L);
	    System.out.print("*");
	}

    }

It usually crushes on the second try before getInstance but sometimes it just hangs there. Is there any fix or workaround because I have integrated the clusterJ api in the Glassfish AS as Java connector adaptor and it is possible to use a session that was created with earlier SessionFactory. If you need any crush dumps let me know will post them.

It appears the problem has been fixed in version clusterj-7.4.6.

The issue is still alive in 7.4.6.
With 3 threads, none of workarounds cited here are working.
I got the JVM crash after 2 minutes of work (800000 rows inserted).
I also tried to get a session in a synchronized block. No way.
I have a simple desktop AMD 2 cores.
My project is a bit fat to be joined and I didn't build a shorter case.
Well, this completely compromizes the use of NDB despite its interest in term of speed with a single thread (more or less equivalent with InnoDB with a database on SSD).