Bug #57037 Schema changes lock up when Cluster is "half-upgraded"
Submitted: 27 Sep 2010 13:46 Modified: 31 Dec 2010 18:46
Reporter: Andrew Morgan Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:mysql-5.1-telco-7.0, mysql-5.1-telco-7.1 OS:Any
Assigned to: CPU Architecture:Any

[27 Sep 2010 13:46] Andrew Morgan
Description:
I have a Cluster running on 7.0.13. I then upgrade one of the data nodes to 7.1.5.

All other nodes stay on 7.0.13.

I then attempt to drop an existing NDB table. That command hangs and if I then try to use the database from another connection, that hangs too.

How to repeat:
7.0.13:
[billy@ws2 my_cluster]$ ndb_mgmd --initial -f conf/config.ini --configdir=/home/billy/my_cluster/conf/
[billy@ws2 my_cluster]$ ndbd -c 127.0.0.1:1186 --initial
2010-09-27 14:19:39 [ndbd] INFO     -- Configuration fetched from '127.0.0.1:1186', generation: 1
[billy@ws2 my_cluster]$ ndbd -c 127.0.0.1:1186 --initial
2010-09-27 14:19:40 [ndbd] INFO     -- Configuration fetched from '127.0.0.1:1186', generation: 1

[billy@ws2 mysql]$ bin/mysql_install_db --datadir=/home/billy/my_cluster/mysqld_data
[billy@ws2 mysql]$ bin/mysql_install_db --datadir=/home/billy/my_cluster/mysqld_data2

[billy@ws2 my_cluster]$ mysql -h 127.0.0.1 -P3306 -u root

mysql> create database clusterdb;
Query OK, 1 row affected (0.07 sec)

mysql> use clusterdb;
Database changed
mysql> create table simples (id int not null primary key) engine=ndb;
100927 14:31:39 [Note] NDB Binlog: CREATE TABLE Event: REPL$clusterdb/simples
Query OK, 0 rows affected (0.86 sec)

mysql> create table simples4 (id int not null primary key, value int) engine=ndb;
100927 14:31:39 [Note] NDB Binlog: CREATE TABLE Event: REPL$clusterdb/simples4
Query OK, 0 rows affected (0.86 sec)

[billy@ws2 my_cluster]$ ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]    2 node(s)
id=3    @127.0.0.1  (mysql-5.1.41 ndb-7.0.13, Nodegroup: 0, Master)
id=4    @127.0.0.1  (mysql-5.1.41 ndb-7.0.13, Nodegroup: 0)

[ndb_mgmd(MGM)]    1 node(s)
id=1    @127.0.0.1  (mysql-5.1.41 ndb-7.0.13)

[mysqld(API)]    2 node(s)
id=50    @127.0.0.1  (mysql-5.1.41 ndb-7.0.13)
id=51 (not connected, accepting connect from any host)

ndb_mgm> 4 stop

[billy@ws2 local]$ rm mysql

# Note that "mysql" was a soft-link to the Cluster 7.0.13 install directory

[billy@ws2 local]$ ln -s mysql-cluster-gpl-7.1.5-linux-x86_64-glibc23 mysql

mysql> insert into simples values (1),(2),(3);
mysql> drop table simples4;
Query OK, 0 rows affected (0.32 sec)

[billy@ws2 my_cluster]$ ndbd -c 127.0.0.1:1186

[billy@ws2 my_cluster]$ mysql -h 127.0.0.1 -P3306 -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.1.41-ndb-7.0.13 Source distribution

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use clusterdb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+---------------------+
| Tables_in_clusterdb |
+---------------------+
| simples             |
| simples2            |
| simples3            |
+---------------------+
3 rows in set (0.00 sec)

mysql> drop table simples3;

# Hangs and appears to lock the database

Suggested fix:
Should be possible to run schema changes on a Cluster where different data nodes are running on different versions.
[27 Sep 2010 13:47] Andrew Morgan
Cluster log

Attachment: ndb_1_cluster.log (application/octet-stream, text), 20.20 KiB.

[27 Sep 2010 13:48] Andrew Morgan
ndb_mgmd log

Attachment: ndb_1_out.log (application/octet-stream, text), 94 bytes.

[27 Sep 2010 13:48] Andrew Morgan
Data node log (7.1.5)

Attachment: ndb_3_out.log (application/octet-stream, text), 7.23 KiB.

[27 Sep 2010 13:49] Andrew Morgan
Data node log (7.1.5)

Attachment: ndb_4_out.log (application/octet-stream, text), 14.35 KiB.

[27 Sep 2010 13:57] Andrew Morgan
I then restarted the management node and mysqld using 7.1.5 - no change.

Finally I restarted the other data node with 7.1.5 and I was able to drop the table.
[31 Dec 2010 18:46] Sveta Smirnova
Verified as described using 2 tables: simples and simples4