Bug #32735 Ndb is unreliable, fails with varying symptoms (platform-specific): debug server
Submitted: 26 Nov 2007 19:40 Modified: 24 Aug 2010 7:39
Reporter: Joerg Bruehe Email Updates:
Status: Won't fix Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:mysql-5.0 OS:MacOS (10.5 (x86: 32 bit))
Assigned to: CPU Architecture:Any
Tags: 5.0.51

[26 Nov 2007 19:40] Joerg Bruehe
Description:
During a build using the 5.0.51 (community) sources,
I had several failures of NDB-related tests.
This bug report only cobers testing the "debug" server,
due to the report size I will file a separate one for the optimized server:

mysqltest                      [ pass ]          24664
ndb_alter_table                [ pass ]          29141
ndb_alter_table2               [ pass ]           3189
ndb_autodiscover               [ fail ]

mysqltest: At line 495: query 'drop database test_only_ndb_tables' failed: 1051: Unknown table 't1'

The result from queries just before the failure was:
< snip >
select * from t1;
a       b
show tables;
Tables_in_test
t1
drop table t1;
use test2;
drop table t2;
drop database test2;
use test;
drop database if exists test_only_ndb_tables;
create database test_only_ndb_tables;
use test_only_ndb_tables;
create table t1 (a int primary key) engine=ndb;
select * from t1;
a
select * from t1;
ERROR HY000: Can't lock file (errno: 157)
use test;
drop database test_only_ndb_tables;

More results from queries before failure can be found in /Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/log/ndb_autodiscover.log

Stopping All Servers
Warning;  Aborted waiting on pid file: '/Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/run/master1.pid' after 70 seconds
Warning;  Aborted waiting on pid file: '/Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/run/master.pid' after 70 seconds
mysql-test-run: WARNING: Forcing kill of process 66476
mysql-test-run: WARNING: Forcing kill of process 66477
Restoring snapshot of databases
Resuming Tests

ndb_autodiscover2              [ fail ]

mysqltest: At line 10: query 'select * from t9 order by a' failed: 1146: Table 'test.t9' doesn't exist

The result from queries just before the failure was:
select * from t9 order by a;

More results from queries before failure can be found in /Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/log/ndb_autodiscover2.log

Stopping All Servers
Restoring snapshot of databases
Resuming Tests

ndb_autodiscover3              [ fail ]

mysqltest: At line 47: query 'create table t2 (a int key) engine=ndbcluster' failed: 157: Could not connect to storage engine

The result from queries just before the failure was:
drop table if exists t1, t2;
create table t1 (a int key) engine=ndbcluster;
begin;
insert into t1 values (1);
insert into t1 values (2);
ERROR HY000: Got temporary error 4025 'Node failure caused abort of transaction' from ndbcluster
commit;
ERROR HY000: Got error 4350 'Transaction already aborted' from ndbcluster
drop table t1;
create table t2 (a int, b int, primary key(a,b)) engine=ndbcluster;
insert into t2 values (1,1),(2,1),(3,1),(4,1),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1);
select * from t2 order by a limit 3;
a       b
1       1
2       1
3       1
create table t2 (a int key) engine=ndbcluster;

More results from queries before failure can be found in /Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/log/ndb_autodiscover3.log

Warnings from just before the error:
Error 1296 Got error 4009 'Cluster Failure' from NDB

Stopping All Servers
Warning;  Aborted waiting on pid file: '/Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/run/master1.pid' after 70 seconds
Warning;  Aborted waiting on pid file: '/Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/run/master.pid' after 70 seconds
mysql-test-run: WARNING: Forcing kill of process 66700
mysql-test-run: WARNING: Forcing kill of process 66701
Restoring snapshot of databases
Resuming Tests

ndb_autodiscover3              [ fail ]

mysqltest: At line 47: query 'create table t2 (a int key) engine=ndbcluster' failed: 157: Could not connect to storage engine

The result from queries just before the failure was:
drop table if exists t1, t2;
create table t1 (a int key) engine=ndbcluster;
begin;
insert into t1 values (1);
insert into t1 values (2);
ERROR HY000: Got temporary error 4025 'Node failure caused abort of transaction' from ndbcluster
commit;
ERROR HY000: Got error 4350 'Transaction already aborted' from ndbcluster
drop table t1;
create table t2 (a int, b int, primary key(a,b)) engine=ndbcluster;
insert into t2 values (1,1),(2,1),(3,1),(4,1),(5,1),(6,1),(7,1),(8,1),(9,1),(10,1);
select * from t2 order by a limit 3;
a       b
1       1
2       1
3       1
create table t2 (a int key) engine=ndbcluster;

More results from queries before failure can be found in /Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/log/ndb_autodiscover3.log

Warnings from just before the error:
Error 1296 Got error 4009 'Cluster Failure' from NDB

Stopping All Servers
Warning;  Aborted waiting on pid file: '/Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/run/master1.pid' after 70 seconds
Warning;  Aborted waiting on pid file: '/Users/mysqldev/tmp-200711150808-5.0.51-30861/xserve-e/test/mysql-debug-5.0.51-osx10.5-x86/mysql-test/var/run/master.pid' after 70 seconds
mysql-test-run: WARNING: Forcing kill of process 66700
mysql-test-run: WARNING: Forcing kill of process 66701
Restoring snapshot of databases
Resuming Tests

Then, many tests pass, but some NDB-related ones get error
   157: Could not connect to storage engine
and the message
Warnings from just before the error:
Error 1296 Got error 4009 'Cluster Failure' from NDB

This happens for tests
ndb_backup_print
ndb_restore_print
rpl_ndb_innodb_trans
strict_autoinc_5ndb

Test "ndb_restore_different_endian_data" encounters bug#31453,
that is a different issue.

How to repeat:
Occurred while running the test suite using the debug server.
[26 Nov 2007 20:00] Joerg Bruehe
See bug#32737 for failures in the optimized server, built from the same sources.