Bug #23334 Automatic Cluster shutdown (Initiated by signal 0. Caused by error 2341)
Submitted: 16 Oct 2006 13:40 Modified: 19 Nov 2006 17:03
Reporter: Roland Bouman Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:5.1.11 OS:Linux (Ubuntu dapper drake)
Assigned to: CPU Architecture:Any
Tags: cluster, crash, error 2341, ndb

[16 Oct 2006 13:40] Roland Bouman
Description:
MySQL Cluster can be reproducible provoked to shutdown all data nodes executing only SQL statements.

The cluster was started with the initial option on a clean system.

How to repeat:
A connection is made to one of the sql nodes using the mysql command line executable. Then, the following SQL statements are executed:

mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> create table test (id int not null auto_increment primary key, ts timestamp, name char(40) not null) engine = ndb;
Query OK, 0 rows affected (1.54 sec)

mysql> insert into test (name) values (sha1(rand()));
Query OK, 1 row affected (0.02 sec)

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 1 row affected (0.05 sec)
Records: 1  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 2 rows affected (0.05 sec)
Records: 2  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 4 rows affected (0.03 sec)
Records: 4  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 8 rows affected (0.05 sec)
Records: 8  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 16 rows affected (0.04 sec)
Records: 16  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 32 rows affected (0.06 sec)
Records: 32  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 64 rows affected (0.05 sec)
Records: 64  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 128 rows affected (0.06 sec)
Records: 128  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 256 rows affected (0.04 sec)
Records: 256  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 512 rows affected (0.08 sec)
Records: 512  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 1024 rows affected (0.12 sec)
Records: 1024  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 2048 rows affected (0.24 sec)
Records: 2048  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 4096 rows affected (0.43 sec)
Records: 4096  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 8192 rows affected (0.83 sec)
Records: 8192  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 16384 rows affected (1.57 sec)
Records: 16384  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 32768 rows affected (3.39 sec)
Records: 32768  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
Query OK, 65536 rows affected (7.02 sec)
Records: 65536  Duplicates: 0  Warnings: 0

mysql> insert into test (name) select sha1(rand()) from test;
ERROR 1297 (HY000): Got temporary error 4010 'Node failure caused abort of transaction' from NDBCLUSTER
mysql>

Suggested fix:
Don't crash
[16 Oct 2006 13:41] Roland Bouman
The cluster log file. This contains only the info from the startup (with --initial) up to the crash.

Attachment: cluster-log (application/octet-stream, text), 9.01 KiB.

[16 Oct 2006 13:42] Roland Bouman
The cluster configuration file

Attachment: config.ini (application/octet-stream, text), 585 bytes.

[16 Oct 2006 13:42] Roland Bouman
Log from the management node

Attachment: ndb_1_out.log (text/x-log), 79 bytes.

[16 Oct 2006 13:43] Roland Bouman
error log from the first datanode

Attachment: ndb_2_error.log (text/x-log), 568 bytes.

[16 Oct 2006 13:44] Roland Bouman
Log from the first datanode

Attachment: ndb_2_out.log (text/x-log), 2.32 KiB.

[16 Oct 2006 13:48] Roland Bouman
second data node error log

Attachment: ndb_3_error.log (text/x-log), 568 bytes.

[16 Oct 2006 13:49] Roland Bouman
log of the second data node

Attachment: ndb_3_out.log (text/x-log), 2.32 KiB.

[16 Oct 2006 13:54] Roland Bouman
Trace log for the second data node

Attachment: ndb_3_trace.log.1.zip (application/zip, text), 182.95 KiB.

[16 Oct 2006 13:55] Roland Bouman
Trace log for the first datanode

Attachment: ndb_2_trace.log.1.zip (application/zip, text), 195.02 KiB.

[17 Oct 2006 12:17] Hartmut Holzgraefe
i managed to crash both the ndbds and my mysqld on the 64k -> 128k rows step, 
but with different error messages (will upload files later)
[19 Oct 2006 12:55] Jonas Oreland
NoOfFragmentLogfiles=1 does not work (and has been prohibited)
This is cause of Rolands crash

Reason for Harmuts is yet unknown.
Changing to "Need feedback"
[19 Oct 2006 14:38] Roland Bouman
Thanks Jonas.

I have verified that 5.1.12 prevents specifying less than 3 log files.
[19 Oct 2006 17:03] Jonas Oreland
resetting to need feedback,
since rolands bug is already fixed...

however hartmut is still unknown
[20 Nov 2006 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".