Bug #19144 (ASP/D) causes 2nd MySQLD process to become out of sync and corrupt
Submitted: 17 Apr 2006 16:56 Modified: 26 Nov 2007 21:29
Reporter: Jonathan Miller Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:5.1.10 OS:Linux (Linux 32 Bit OS)
Assigned to: Jonathan Miller CPU Architecture:Any

[17 Apr 2006 16:56] Jonathan Miller 
Description:
 Automatic Schema Propagation/Distribution (ASP/D) is suppose to keep other MySQLD processes up to data with schema changes that happen inside the database.  Having 2 cid_ndb_dd.perl scripts running against one MySQLD process cause the second process to become out of sysnc and to corrupt.

060417 18:43:08 [Note] Starting MySQL Cluster Binlog Thread
060417 18:43:08 [Note] /home/ndbdev/jmiller/builds/libexec/mysqld: ready for connections.
Version: '5.1.10-beta-log'  socket: '/tmp/mysql3306.sock'  port: 3306  Source distribution
060417 18:43:08 [ERROR] NDB: DROP DATABASE TESTER_2: error Can't drop database 'TESTER_2'; database doesn't exist 1008 1 1
060417 18:43:10 [ERROR] NDB: DROP DATABASE TESTER: error Can't drop database 'TESTER'; database doesn't exist 1008 1 1
060417 18:43:22 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:43:24 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:43:38 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:43:40 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:44:07 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:44:09 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:44:21 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:44:25 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:44:38 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:44:41 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:44:53 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:44:55 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:45:33 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:45:34 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:45:46 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:45:48 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:46:16 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:46:17 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:46:29 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:46:31 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:46:57 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:46:59 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:47:11 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:47:12 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:47:24 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:47:26 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:47:39 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:47:41 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:47:53 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:47:56 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:48:10 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:48:11 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:48:24 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:48:26 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:49:21 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:49:23 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:49:35 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:49:36 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:49:48 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:49:50 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:50:02 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:50:04 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:50:16 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:50:18 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:50:30 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:50:32 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:50:45 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:50:49 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:51:03 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:51:07 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:52:03 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1
060417 18:52:05 [ERROR] NDB: CREATE DATABASE TESTER_2: error Can't create database 'TESTER_2'; database exists 1007 1 1
060417 18:52:18 [ERROR] NDB: DROP DATABASE TESTER: error Unknown table 't1' 1051 1 1
060417 18:52:22 [ERROR] NDB: CREATE DATABASE TESTER: error Can't create database 'TESTER'; database exists 1007 1 1
060417 18:52:33 [ERROR] NDB: DROP DATABASE TESTER_2: error Unknown table 't1' 1051 1 1

How to repeat:
Start a cluster with 2 MySQLD processes.
in one of the mysqld processes run the 2 following commands.
(run in 2 seperate prompts.)

$>perl cid_ndb_dd.pl -ho hostname --port 3306 -u root
$>perl cid_ndb_dd.pl -ho hostname --port 3306 -u root -d TESTER_2
[17 Apr 2006 17:00] Jonathan Miller 
After stopping the test gracefully:

MySQLD #1
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cluster            |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)

MySQLD#2

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| TESTER             |
| cluster            |
| mysql              |
| test               |
+--------------------+
5 rows in set (0.00 sec)

mysql> use TESTER
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+------------------+
| Tables_in_TESTER |
+------------------+
| t1               |
+------------------+
1 row in set (0.01 sec)

mysql> drop table t1;
ERROR 1051 (42S02): Unknown table 't1'
mysql>

mysql> DROP DATABASE TESTER;
Query OK, 0 rows affected (0.61 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cluster            |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)
[17 Apr 2006 17:16] Jonathan Miller 
Error message from MySQLD #1 after manual drop done on MySQLD #2
060417 19:00:03 [ERROR] NDB: DROP DATABASE TESTER: error Can't drop database 'TESTER'; database doesn't exist 1008 1 1
[14 Jun 2006 10:11] Tomas Ulin 
this is a patch that fixes this... lot's of reviews to make it happen though...

Attachment: bug19144.patch (text/x-patch), 14.25 KiB.
[4 Jul 2007 6:06] Calvin Sun 
Assign back to Tomas based on his comments: 

"Likely this patch is totally outdated. and a new would have to be made.

however there is a bugfix for 21074 and some other bug in runtime that may make the need for this bugfix obsolete, I will have to check at some point. "
[26 Nov 2007 21:29] Jonathan Miller 
I retested with mysql-5.1-telco-6.2. No issues found during this testing, so I am closing this issue report.

Best wishes,
/Jeb