MySQL Bugs: #43857: Cluster crash during heavy load

Bug #43857	Cluster crash during heavy load
Submitted:	25 Mar 2009 13:05	Modified:	28 May 2009 8:03
Reporter:	Johan Jerrang	Email Updates:
Status:	Verified	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	mysql-5.1-telco-7.0	OS:	Solaris (SunOS 5.10 Generic_138888-01 sun4v sparc SUNW,Sun-Fire-T200)
Assigned to:		CPU Architecture:	Any
Tags:	mysql-5.1-telco-7.0.4

Description:
After inserting about 1M rows into a fairly large table (around 45 attributes) using 7.0.4 the cluster crashes.

The setup is 2 T2000 machines each running 1 mysqld and 1 ndbmtd. I have a java program doing inserts to both mysql servers with 32 threads per server using Connector/J.

The table uses disk based storage with 2 undo files of 256M each and 4 datafiles of 2048M each.

From the error log:
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Node 3 killed this node because GCP stop was detected
Error object: NDBCNTR (Line: 260) 0x0000000a
Program: /usr/local/mysql/libexec/ndbmtd
Pid: 18435
Trace: /data/mysql-cluster//ndb_3_trace.log.1 /data/mysql-cluster//ndb_3_trace.log.1_t1 /data

The ndb_3_out.log also reports:
/data/mysql-cluster/record_1.dat Failed to set DIRECTIO_ON errno: 25
However this is probably not related. We are running ZFS.

How to repeat:
Insert ~1 million rows into a large table.

Trace and log files

Attachment: ndb_error_report_20090325134711.tar.bz2 (application/octet-stream, text), 299.66 KiB.

http://bugs.mysql.com/bug.php?id=43882
http://bugs.mysql.com/bug.php?id=39498
http://bugs.mysql.com/bug.php?id=44499

Just tried with 7.0.5, the problem still exists.