Bug #46494 can't alter/create table after an upgrade from ndb 6.3.23 to ndb 7.0.6
Submitted: 31 Jul 2009 13:30 Modified: 5 Oct 2009 23:04
Reporter: Cyril SCETBON Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:mysql-5.1.34-telco-7.0.6 OS:Any
Assigned to: Jonas Oreland CPU Architecture:Any
Tags: DDL, ndb, online, upgrade

[31 Jul 2009 13:30] Cyril SCETBON
Description:
I tried the method from http://johanandersson.blogspot.com/2009/06/how-to-upgrade-from-63-to-70.html to upgrade my ndb-6.3.23 cluster to ndb-7.0.6 (as specified in the documentation) and everything worked. However I can no longer create/alter a table without getting this error :

mysql> create table test(id int) engine=ndb;
ERROR 1005 (HY000): Can't create table 'mydb.test' (errno: 140)
mysql> show warnings;
+-------+------+----------------------------------------------------------+
| Level | Code | Message |
+-------+------+----------------------------------------------------------+
| Error | 1296 | Got error 755 'Invalid tablespace' from NDB |
| Error | 1005 | Can't create table 'mydb.test' (errno: 140) |
+-------+------+----------------------------------------------------------+
2 rows in set (0,00 sec)

And I can no longer backup data with ndb_mgm :

d01:~# ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> start backup;
Connected to Management Server at: XX.XX.XX.XX:1186
Waiting for completed, this may take several minutes
Node 3: Backup request from 2 failed to start. Error: 1304
Backup failed
* 3001: Could not start backup
* Sequence failure: Permanent error: Internal error
ndb_mgm> exit

The only way to fix it was to dump data (mysqldump), restart nodes with --initial and to reload the data. That's really a good choice for a 24/7 service.

How to repeat:
- start with a ndb-6.3.23 cluster of 2 nodes with 1 table and some values inserted
- upgrade the cluster using the procedure found at http://johanandersson.blogspot.com/2009/06/how-to-upgrade-from-63-to-70.html
- try to add a new table in the cluster
[31 Jul 2009 13:58] Geert Vanderkelen
Oh I wish I could say  I can't repeat.. but yes..

Verified upgrading from 6.3.23 and 6.3.25 to 7.0.6: one can not create new tables.
[31 Jul 2009 15:22] Geert Vanderkelen
Verified also using 6.3.26(bzr) upgrade to 7.0.7(bzr).
[1 Aug 2009 20:30] Cyril SCETBON
not a really good news :(
[4 Aug 2009 10:54] Robert Klikics
Same problem here! Any fix or workaround available?
[4 Aug 2009 11:01] Robert Klikics
Additional info:

I DID NOT upgrade the cluster. I made a clean dump-import from 6.3.17 to 7.0.6 a few weeks ago. But the same problem appears here!
[4 Aug 2009 12:59] Cyril SCETBON
we had no problem when we did a dump from ndb-6.3 to ndb-7.0
Why would a dump/load cause this issue ?? weird..
[4 Aug 2009 13:29] Robert Klikics
Because I imported the dump a few weeks ago, it seems not to be an bug only caused by upgrading from 6.xx but a major bug in 7.0.6
[4 Aug 2009 14:05] Robert Klikics
A rolling restart with --initial did NOT fix the issue.
I still can not create tables or ALTER them. The mysqld's have also been restarted.

Because this is really bad, I hope there will ne a fix out asap!
[4 Aug 2009 14:12] Cyril SCETBON
you should provide the log/trace/output files
[4 Aug 2009 14:18] Robert Klikics
There is nothing to found within the logs :/
[5 Aug 2009 9:56] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/80130

3005 Jonas Oreland	2009-08-05
      ndb - bug#46494 - test
        Add ability to do post upgrade verification
        This is done by restarting test-program binary in new version
        with the testcase being altered "X" becomes "X--post-upgrade"
     @ storage/ndb/test/run-test/command.cpp
        only append new value, don't clear entire env
     @ storage/ndb/test/run-test/db.cpp
        use "correct" variable
     @ storage/ndb/test/run-test/setup.cpp
        put atrt process id in environment
[5 Aug 2009 10:51] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/80134

2959 Jonas Oreland	2009-08-05
      ndb - bug#46494
        Problem was that after an upgrade from 6.3, no default hashmap existed
        Patch fixes this by always (silently) adding a "create default hashmap if not exists" to each
          create table operation (which works perfectly thx to schema transactions)
      
        Test is committed to 6.3, which as a post-upgrade-test will drop/recreate the table
     @ storage/ndb/include/kernel/signaldata/CreateTable.hpp
        fix unique error code for invalud hashmap
     @ storage/ndb/src/common/debugger/signaldata/SignalNames.cpp
        add signal names
     @ storage/ndb/src/kernel/blocks/dbdict/Dbdict.cpp
        implement
        - CreateHashMapReq::CreateDefault
        - CreateHashMapReq::CreateIfNotExists
     @ storage/ndb/src/ndbapi/NdbDictionaryImpl.cpp
        always add a "create default hashmap if not exists" to a create-table
          if it is hashmap-partitioned and has not specified a hashmap
     @ storage/ndb/src/ndbapi/ndberror.c
        new error code
[5 Aug 2009 12:03] Bugs System
Pushed into 5.1.35-ndb-7.0.7 (revid:jonas@mysql.com-20090805110534-ovymemsb130twcrr) (version source revid:jonas@mysql.com-20090805104856-1dn1t3l1r1gkcyom) (merge vers: 5.1.35-ndb-7.0.7) (pib:11)
[5 Aug 2009 13:13] Robert Klikics
Thanks!

Is there already a release date for 7.0.7?
[6 Aug 2009 8:55] Jon Stephens
Documented bugfix in the NDB-7.0.7 changelog as follows:

      Following an upgrade from MySQL Cluster NDB 6.3.x to MySQL Cluster NDB
      7.0.6, DDL and backup operations failed.

Also noted issue in Upgrades section, updated error code tables with new hashmap error code.
[5 Oct 2009 23:04] Cyril SCETBON
why did we get the same error as we already did a start-initial ?
after having done a start-initial there's no more resident data from 6.3, isn't it ? Why upgrading to 7.0.7 should resolve this issue if it can't be related to our migration since we did a start-initial ?

thanks