MySQL Bugs: #23334: Automatic Cluster shutdown (Initiated by signal 0. Caused by error 2341)

Bug #23334	Automatic Cluster shutdown (Initiated by signal 0. Caused by error 2341)
Submitted:	16 Oct 2006 13:40	Modified:	19 Nov 2006 17:03
Reporter:	Roland Bouman	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	5.1.11	OS:	Linux (Ubuntu dapper drake)
Assigned to:		CPU Architecture:	Any
Tags:	cluster, crash, error 2341, ndb

Description:
MySQL Cluster can be reproducible provoked to shutdown all data nodes executing only SQL statements.

The cluster was started with the initial option on a clean system.

How to repeat:
A connection is made to one of the sql nodes using the mysql command line executable. Then, the following SQL statements are executed:

mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> create table test (id int not null auto_increment primary key, ts timestamp, name char(40) not null) engine = ndb;
Query OK, 0 rows affected (1.54 sec)

mysql> insert into test (name) values (sha1(rand()));
Query OK, 1 row affected (0.02 sec)

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 1 row affected (0.05 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 2 rows affected (0.05 sec)
Records: 2  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 4 rows affected (0.03 sec)
Records: 4  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 8 rows affected (0.05 sec)
Records: 8  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 16 rows affected (0.04 sec)
Records: 16  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 32 rows affected (0.06 sec)
Records: 32  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 64 rows affected (0.05 sec)
Records: 64  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 128 rows affected (0.06 sec)
Records: 128  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 256 rows affected (0.04 sec)
Records: 256  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 512 rows affected (0.08 sec)
Records: 512  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 1024 rows affected (0.12 sec)
Records: 1024  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 2048 rows affected (0.24 sec)
Records: 2048  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 4096 rows affected (0.43 sec)
Records: 4096  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 8192 rows affected (0.83 sec)
Records: 8192  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 16384 rows affected (1.57 sec)
Records: 16384  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 32768 rows affected (3.39 sec)
Records: 32768  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 65536 rows affected (7.02 sec)
Records: 65536  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
ERROR 1297 (HY000): Got temporary error 4010 'Node failure caused abort of transaction' from NDBCLUSTER
mysql>

Suggested fix:
Don't crash

The cluster log file. This contains only the info from the startup (with --initial) up to the crash.

Attachment: cluster-log (application/octet-stream, text), 9.01 KiB.

The cluster configuration file

Attachment: config.ini (application/octet-stream, text), 585 bytes.

Log from the management node

Attachment: ndb_1_out.log (text/x-log), 79 bytes.

error log from the first datanode

Attachment: ndb_2_error.log (text/x-log), 568 bytes.

Log from the first datanode

Attachment: ndb_2_out.log (text/x-log), 2.32 KiB.

second data node error log

Attachment: ndb_3_error.log (text/x-log), 568 bytes.

log of the second data node

Attachment: ndb_3_out.log (text/x-log), 2.32 KiB.

Trace log for the second data node

Attachment: ndb_3_trace.log.1.zip (application/zip, text), 182.95 KiB.

Trace log for the first datanode

Attachment: ndb_2_trace.log.1.zip (application/zip, text), 195.02 KiB.

i managed to crash both the ndbds and my mysqld on the 64k -> 128k rows step, 
but with different error messages (will upload files later)

NoOfFragmentLogfiles=1 does not work (and has been prohibited)
This is cause of Rolands crash

Reason for Harmuts is yet unknown.
Changing to "Need feedback"

Thanks Jonas.

I have verified that 5.1.12 prevents specifying less than 3 log files.

resetting to need feedback,
since rolands bug is already fixed...

however hartmut is still unknown

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".