Bug #62116 Complete cluster crash during INSERT statement

Description:
We configured a MySQL Cluster 7.1.15 with 2 data nodes, 2 mgm nodes and serveral sql nodes, as per the config.ini you can find into the attached ndb_error_reporter generated archive.

After starting both the data nodes with --initial, we started a single sqlnode (172.16.1.142), we created a new database, a new table in that empty database and we issued an INSERT statement.

During that INSERT both data nodes crashed reporting this error:

Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

Please, find attached the ndb_error_reporter generated archive containing all the error logs.

How to repeat:
Configure a new MySQL Cluster 7.1.15 as per the config.ini you can find into the attached ndb_error_reporter generated archive.

Start both the configured management nodes and both the data nodes with --initial.

Start the sql node 172.16.1.142 and from its mysqld, issue the followings:
> CREATE DATABASE bug;
> USE bug;
> CREATE TABLE `t1` (
  `a` bigint(20) DEFAULT NULL,
  `b` char(100) DEFAULT NULL,
  `c` int(64) DEFAULT NULL,
  `d` varchar(1000) DEFAULT NULL,
  KEY `a` (`a`),
  KEY `b` (`b`),
  KEY `c` (`c`),
  KEY `d` (`d`),
  KEY `a_2` (`a`,`b`),
  KEY `a_3` (`a`,`c`),
  KEY `a_4` (`a`,`d`),
  KEY `b_2` (`b`,`c`)
) ENGINE=ndbcluster DEFAULT CHARSET=latin1;

> INSERT INTO bug.t1 VALUES (1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2');

During this INSERT, both the data nodes crash.

Suggested fix:
None

ndb_error_reporter generated archive

Attachment: ndb_error_report_20110808165350.tar.bz2 (application/x-bzip2, text), 367.61 KiB.

Additional tests reported below.

Same cluster set up and database/table creation, INSERT working:

> INSERT INTO bug.t1 VALUES (1,'Test',2,'Test2');
Query OK, 1 row affected (0.01 sec)

> TRUNCATE bug.t1;
Query OK, 0 rows affected (0.27 sec

> INSERT INTO bug.t1 VALUES (1,'Test',2,'Test2'),(1,'Test',2,'Test2');
Query OK, 2 rows affected (0.00 sec)
Records: 2  Duplicates: 0  Warnings: 0

> TRUNCATE bug.t1;
Query OK, 0 rows affected (0.31 sec)

> INSERT INTO bug.t1 VALUES (1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2');
Query OK, 3 rows affected (0.01 sec)
Records: 3  Duplicates: 0  Warnings: 0

INSERT NOT WORKING:

> TRUNCATE bug.t1;
Query OK, 0 rows affected (0.34 sec)

> INSERT INTO bug.t1 VALUES (1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2');
ERROR 1297 (HY000): Got temporary error 4010 'Node failure caused abort of transaction' from NDBCLUSTER

Both data nodes crash.

Trying with more rows, still cause both data nodes crash:

mysql> INSERT INTO bug.t1 VALUES (1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2'),(1,'Test',2,'Test2');
ERROR 1297 (HY000): Got temporary error 4010 'Node failure caused abort of transaction' from NDBCLUSTER

and so on...

ndb_error_reporter generated archive - All nodes at 7.1.15

Attachment: ndb_error_report_20110808172339.tar.bz2 (application/x-bzip2, text), 351.58 KiB.

First attached ndb_error_reporter generated archive was related to test performed with sql node at version 7.1.8.

Please, find attached the proper ndb_error_reporter generated archive related to tests with all nodes at 7.1.15.

The problem is still there, with both version of sql nodes.

Hi Fabio,

I tried with your exact config, create-table and insert and could not reproduce.
Have you built the cluster your self, or downloaded binaries ?

/Jonas

Hi Jonas,

I downloaded the RHEL RPMs: MySQL-Cluster-gpl-<package>-7.1.15-1.rhel5.x86_64.rpm
Installed on a RHEL 5.4 64bit hosted on a VMware ESXi 4.0 Virtual Machine.

I'll try to re-create the VMs .

Anyway, for your information, on the same server infrastructure with all nodes at version 7.1.8 there's no crash or problem.
I tried version 7.1.8 because we have a cluster at that version in production and we're testing the new 7.1.15 before upgrading.

Thanks for you support,
Fabio

we found it,
and will release a new version asap

/Jonas

Fix

Attachment: bug.12873640.patch (application/octet-stream, text), 1.41 KiB.