Bug #21875 "create table" fails in NDB with error 156
Submitted: 28 Aug 2006 14:09 Modified: 28 Sep 2006 5:09
Reporter: Joerg Bruehe Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:5.0.23 and up OS:Linux (Linux (ia64))
Assigned to: Martin Skold CPU Architecture:Any

[28 Aug 2006 14:09] Joerg Bruehe
Description:
This test failure started to appear with 5.0.23, unchanged in 5.0.24 and 24a.

It is specific to "rx2620b" (Linux, ia64, using icc to compile).

It appears in both "max" and "cluster" builds in some NDB tests.

The test suite aborts in "ps_7ndb" in both the "max" and the "cluster" run
("waitNodeState(STARTED, -1) timeout after 121 attemps"),
so I cannot tell whether the later tests involving NDB would show related issues.
This abort is also identical in 5.0.23, 24, and 24a.

These are the failure reports (quoted from 5.0.24a):

=====

ndb_alter_table                [ fail ]

Errors are (from /home/mysqldev/rx2620b-icc-glibc23/test/mysql-max-5.0.24a-linux-ia64-icc-glibc23/mysql-test/var/log/mysqltest-time) :
mysqltest: At line 125: query 'alter table  t1 modify column a int not null auto_increment' failed: 1005: Can't create table './test/#sql-7c6f_4.frm' (errno: 156)
(the last lines may be the most important ones)
Result from queries before failure can be found in r/ndb_alter_table.log

Killing Possible Leftover Processes
Restoring snapshot of databases
Resuming Tests

ndb_alter_table2               [ pass ]           4050
ndb_autodiscover               [ fail ]

Errors are (from /home/mysqldev/rx2620b-icc-glibc23/test/mysql-max-5.0.24a-linux-ia64-icc-glibc23/mysql-test/var/log/mysqltest-time) :
mysqltest: At line 440: query 'ALTER TABLE t6 ADD COLUMN adress char(255) FIRST' failed: 1005: Can't create table './test/#sql-7d3c_9.frm' (errno: 156)
(the last lines may be the most important ones)
Result from queries before failure can be found in r/ndb_autodiscover.log

Killing Possible Leftover Processes
Restoring snapshot of databases
Resuming Tests

.....

ndb_cache2                     [ fail ]

Errors are (from /home/mysqldev/rx2620b-icc-glibc23/test/mysql-max-5.0.24a-linux-ia64-icc-glibc23/mysql-test/var/log/mysqltest-time) :
mysqltest: At line 25: query 'CREATE TABLE t2 (
pk int not null primary key,
a2 int,
b2 int not null
) ENGINE=ndb' failed: 1005: Can't create table './test/t2.frm' (errno: 156)
(the last lines may be the most important ones)
Result from queries before failure can be found in r/ndb_cache2.log

Killing Possible Leftover Processes
Restoring snapshot of databases
Resuming Tests

.....

ndb_index_ordered              [ fail ]

Errors are (from /home/mysqldev/rx2620b-icc-glibc23/test/mysql-max-5.0.24a-linux-ia64-icc-glibc23/mysql-test/var/log/mysqltest-time) :
mysqltest: At line 95: query 'create index a1 on t1 (b, c)' failed: 1005: Can't create table './test/#sql-2ca_c.frm' (errno: 156)
(the last lines may be the most important ones)
Result from queries before failure can be found in r/ndb_index_ordered.log

Killing Possible Leftover Processes
Restoring snapshot of databases
Resuming Tests

ndb_index_unique               [ fail ]

Errors are (from /home/mysqldev/rx2620b-icc-glibc23/test/mysql-max-5.0.24a-linux-ia64-icc-glibc23/mysql-test/var/log/mysqltest-time) :
mysqltest: At line 169: query 'CREATE TABLE t2 (
cid bigint(20) unsigned NOT NULL auto_increment,
cap varchar(255) NOT NULL default '',
PRIMARY KEY  (cid),
UNIQUE KEY (cid, cap)
) engine=ndbcluster' failed: 1005: Can't create table './test/t2.frm' (errno: 156)
(the last lines may be the most important ones)
Result from queries before failure can be found in r/ndb_index_unique.log

Killing Possible Leftover Processes
Restoring snapshot of databases
Resuming Tests

.....

ndb_replace                    [ fail ]

Errors are (from /home/mysqldev/rx2620b-icc-glibc23/test/mysql-max-5.0.24a-linux-ia64-icc-glibc23/mysql-test/var/log/mysqltest-time) :
mysqltest: At line 44: query 'CREATE TABLE t2 (a INT(11) NOT NULL,
b INT(11) NOT NULL,
c INT(11) NOT NULL,
x TEXT,
y TEXT,
z TEXT,
id INT(10) unsigned NOT NULL AUTO_INCREMENT,
i INT(11) DEFAULT NULL,
PRIMARY KEY (id),
UNIQUE KEY a (a,b,c)
) ENGINE=ndbcluster' failed: 1005: Can't create table './test/t2.frm' (errno: 156)
(the last lines may be the most important ones)
Result from queries before failure can be found in r/ndb_replace.log

Killing Possible Leftover Processes
Restoring snapshot of databases
Resuming Tests

=====

How to repeat:
Run the test suite.
[28 Aug 2006 14:13] Jonas Oreland
unless you have changed icc version this is 99% a duplicate of
http://bugs.mysql.com/bug.php?id=17621
[28 Aug 2006 14:29] Joerg Bruehe
Sorry, my search had not led me to that entry.

The compiler was upgraded:

[mysqldev@rx2620b mysqldev]$ . path-for-rx2620b
[mysqldev@rx2620b mysqldev]$ icc -V
Intel(R) C Itanium(R) Compiler for Itanium(R)-based applications
Version 9.1    Build 20060523 Package ID: l_cc_c_9.1.039
Copyright (C) 1985-2006 Intel Corporation.  All rights reserved.

This is the "configure" line (from "max"):

CC=icc CFLAGS="-O3 -unroll2 -ip -mp -restrict" CXX=icpc CXXFLAGS="-O3 -unroll2 -ip -mp -restrict" LDFLAGS="-static-libcxa -i-static"  ./configure --prefix=/usr/local/mysql --localstatedir=/usr/local/mysql/data --libexecdir=/usr/local/mysql/bin --with-comment="MySQL Community Edition - Experimental (GPL)" --with-server-suffix="-max" --enable-thread-safe-client --enable-local-infile  --with-pic --with-fast-mutexes --disable-shared --with-zlib-dir=bundled --with-big-tables --with-readline --with-archive-storage-engine --with-blackhole-storage-engine --with-ndbcluster --with-csv-storage-engine --with-example-storage-engine --with-federated-storage-engine --with-innodb --with-extra-charsets=all

If dropping "-ip" had helped you, we can of course do a run with the new compiler without that option.
[29 Aug 2006 13:03] Jonas Oreland
Is it correct that it only fails with icc ?
Then skip icc build for ndb

Assigning to martin so he can agree/disagree.
[20 Sep 2006 9:51] Mads Martin Joergensen
Upgrade of icc "fixed" this, so this is not a bug in our code, but was a bug in the icc we were using.

Setting to documenting, just in case we want to document this in some way.
[25 Sep 2006 8:10] Jon Stephens
I think that this should be documented in the 5.0/5.1 Manual under Installation/Building From Source, but first a stupid question - what were the "old" and "new" icc versions? (I've never used icc myself and am not sure where to get this info.)

Also, were any 4.1 versions of the server affected by this problem?

Thanks!
[25 Sep 2006 8:39] Joerg Bruehe
About 4.1:  I cannot tell for sure.
We did not notice this specific problem in 4.1.20 or 4.1.21,
but I cannot tell the reason (code differences so that this compiler bug was not triggered, or test differences so that it was not shown, or ...)

About "icc" versions:
The "new" vesion is called 9.1.043,
and the compiler shows this output:
   mysqldev@nocona:~> icc --version
   icc (ICC) 9.1 20060818
   Copyright (C) 1985-2006 Intel Corporation.  All rights reserved.

The previous version, which had the problem, was 9.1.039, identifying as 
   icc (ICC) 9.1 20060523

Note that all this is specific to "ia64" CPUs,
the problem never occurred on "x86" or "x86_64".
[28 Sep 2006 5:09] Jon Stephens
Thank you for your bug report. This issue has been addressed in the documentation. The updated documentation will appear on our website shortly, and will be included in the next release of the relevant products.

Added a note about this issue to the Porting section in the 5.0 and 5.1 Manual.

Thanks for the info.