MySQL Bugs: #46494: can't alter/create table after an upgrade from ndb 6.3.23 to ndb 7.0.6

Bug #46494	can't alter/create table after an upgrade from ndb 6.3.23 to ndb 7.0.6
Submitted:	31 Jul 2009 13:30	Modified:	5 Oct 2009 23:04
Reporter:	Cyril SCETBON	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	mysql-5.1.34-telco-7.0.6	OS:	Any
Assigned to:	Jonas Oreland	CPU Architecture:	Any
Tags:	DDL, ndb, online, upgrade

Description:
I tried the method from http://johanandersson.blogspot.com/2009/06/how-to-upgrade-from-63-to-70.html to upgrade my ndb-6.3.23 cluster to ndb-7.0.6 (as specified in the documentation) and everything worked. However I can no longer create/alter a table without getting this error :

mysql> create table test(id int) engine=ndb;
ERROR 1005 (HY000): Can't create table 'mydb.test' (errno: 140)
mysql> show warnings;
+-------+------+----------------------------------------------------------+
| Level | Code | Message |
+-------+------+----------------------------------------------------------+
| Error | 1296 | Got error 755 'Invalid tablespace' from NDB |
| Error | 1005 | Can't create table 'mydb.test' (errno: 140) |
+-------+------+----------------------------------------------------------+
2 rows in set (0,00 sec)

And I can no longer backup data with ndb_mgm :

d01:~# ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> start backup;
Connected to Management Server at: XX.XX.XX.XX:1186
Waiting for completed, this may take several minutes
Node 3: Backup request from 2 failed to start. Error: 1304
Backup failed
* 3001: Could not start backup
* Sequence failure: Permanent error: Internal error
ndb_mgm> exit

The only way to fix it was to dump data (mysqldump), restart nodes with --initial and to reload the data. That's really a good choice for a 24/7 service.

How to repeat:
- start with a ndb-6.3.23 cluster of 2 nodes with 1 table and some values inserted
- upgrade the cluster using the procedure found at http://johanandersson.blogspot.com/2009/06/how-to-upgrade-from-63-to-70.html
- try to add a new table in the cluster

Oh I wish I could say  I can't repeat.. but yes..

Verified upgrading from 6.3.23 and 6.3.25 to 7.0.6: one can not create new tables.

Verified also using 6.3.26(bzr) upgrade to 7.0.7(bzr).

not a really good news :(

Same problem here! Any fix or workaround available?

Additional info:

I DID NOT upgrade the cluster. I made a clean dump-import from 6.3.17 to 7.0.6 a few weeks ago. But the same problem appears here!

we had no problem when we did a dump from ndb-6.3 to ndb-7.0
Why would a dump/load cause this issue ?? weird..

Because I imported the dump a few weeks ago, it seems not to be an bug only caused by upgrading from 6.xx but a major bug in 7.0.6

A rolling restart with --initial did NOT fix the issue.
I still can not create tables or ALTER them. The mysqld's have also been restarted.

Because this is really bad, I hope there will ne a fix out asap!

you should provide the log/trace/output files

There is nothing to found within the logs :/

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/80130

3005 Jonas Oreland	2009-08-05
      ndb - bug#46494 - test
        Add ability to do post upgrade verification
        This is done by restarting test-program binary in new version
        with the testcase being altered "X" becomes "X--post-upgrade"
     @ storage/ndb/test/run-test/command.cpp
        only append new value, don't clear entire env
     @ storage/ndb/test/run-test/db.cpp
        use "correct" variable
     @ storage/ndb/test/run-test/setup.cpp
        put atrt process id in environment

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/80134

2959 Jonas Oreland	2009-08-05
      ndb - bug#46494
        Problem was that after an upgrade from 6.3, no default hashmap existed
        Patch fixes this by always (silently) adding a "create default hashmap if not exists" to each
          create table operation (which works perfectly thx to schema transactions)
      
        Test is committed to 6.3, which as a post-upgrade-test will drop/recreate the table
     @ storage/ndb/include/kernel/signaldata/CreateTable.hpp
        fix unique error code for invalud hashmap
     @ storage/ndb/src/common/debugger/signaldata/SignalNames.cpp
        add signal names
     @ storage/ndb/src/kernel/blocks/dbdict/Dbdict.cpp
        implement
        - CreateHashMapReq::CreateDefault
        - CreateHashMapReq::CreateIfNotExists
     @ storage/ndb/src/ndbapi/NdbDictionaryImpl.cpp
        always add a "create default hashmap if not exists" to a create-table
          if it is hashmap-partitioned and has not specified a hashmap
     @ storage/ndb/src/ndbapi/ndberror.c
        new error code

Pushed into 5.1.35-ndb-7.0.7 (revid:jonas@mysql.com-20090805110534-ovymemsb130twcrr) (version source revid:jonas@mysql.com-20090805104856-1dn1t3l1r1gkcyom) (merge vers: 5.1.35-ndb-7.0.7) (pib:11)

Thanks!

Is there already a release date for 7.0.7?

Documented bugfix in the NDB-7.0.7 changelog as follows:

      Following an upgrade from MySQL Cluster NDB 6.3.x to MySQL Cluster NDB
      7.0.6, DDL and backup operations failed.

Also noted issue in Upgrades section, updated error code tables with new hashmap error code.

why did we get the same error as we already did a start-initial ?
after having done a start-initial there's no more resident data from 6.3, isn't it ? Why upgrading to 7.0.7 should resolve this issue if it can't be related to our migration since we did a start-initial ?

thanks