MySQL Bugs: #18575: Cluster crash running 2 stress scripts (cid_ndb

Bug #18575	Cluster crash running 2 stress scripts (cid_ndb_dd2.pl & looper.sh)
Submitted:	28 Mar 2006 14:39	Modified:	24 Apr 2006 16:17
Reporter:	Jonathan Miller	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	5.1.9	OS:	Linux (Linux 32 Bit OS)
Assigned to:	Jonas Oreland	CPU Architecture:	Any

Description:
After the TPC-B load had failed last night due to mysqld process crashing, I left  cid_ndb_dd2.pl & looper.sh running. This morning I found that the cluster had crashed.

Cluster Error Log:
2006-03-28 03:38:27 [MgmSrvr] INFO     -- Node 2: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_blockCommit = 1 c_blockCommitNo = 2 sig->failNo =
2006-03-28 03:38:27 [MgmSrvr] INFO     -- Node 2: Communication to Node 3 closed
2006-03-28 03:38:27 [MgmSrvr] INFO     -- Node 2: Communication to Node 4 closed
2006-03-28 03:38:27 [MgmSrvr] ALERT    -- Node 2: Network partitioning - arbitration required
2006-03-28 03:38:27 [MgmSrvr] INFO     -- Node 2: President restarts arbitration thread [state=7]2006-03-28 03:38:27 [MgmSrvr] INFO     -- Node 1: Node 4 Connected
2006-03-28 03:38:27 [MgmSrvr] ALERT    -- Node 5: Node 4 Disconnected
2006-03-28 03:38:27 [MgmSrvr] INFO     -- Node 5: Possible bug in Dbdih::execBLOCK_COMMIT_ORD c_blockCommit = 1 c_blockCommitNo = 2 sig->failNo =
2006-03-28 03:38:27 [MgmSrvr] INFO     -- Node 5: Communication to Node 3 closed
2006-03-28 03:38:27 [MgmSrvr] INFO     -- Node 5: Communication to Node 4 closed
2006-03-28 03:38:27 [MgmSrvr] ALERT    -- Node 3: Forced node shutdown completed. Initiated by signal 0. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2006-03-28 03:38:27 [MgmSrvr] ALERT    -- Node 2: Arbitration won - positive reply from node 1
2006-03-28 03:52:23 [MgmSrvr] INFO     -- Node 1: Node 2 Connected
2006-03-28 03:52:23 [MgmSrvr] ALERT    -- Node 2: Forced node shutdown completed. Initiated by signal 0. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2006-03-28 03:52:24 [MgmSrvr] INFO     -- Node 1: Node 5 Connected
2006-03-28 03:52:24 [MgmSrvr] ALERT    -- Node 5: Forced node shutdown completed. Initiated by signal 0. Caused by error 2305: 'Arbitrator shutdown, please investigate error(s) on other node(s)(Arbitration error). Temporary error, restart node'.

More to be added

How to repeat:
I will up load the test scripts

Time: Tuesday 28 Mars 2006 - 03:48:58
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: pgman.cpp
Error object: PGMAN (Line: 1601) 0x0000000a
Program: /home/ndbdev/jmiller/builds/libexec/ndbd
Pid: 22658
Trace: /space/run/ndb_2_trace.log.1
Version: Version 5.1.9 (beta)
--------------- Signal ----------------
r.bn: 259 "TSMAN", r.proc: 2, r.sigId: 27845101 gsn: 722 "DROP_FILE_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 2, s.sigId: 27845100 length: 6 trace: 0 #sec: 0 fragInf: 0
 H'00000423 H'00fa0002 H'00000002 H'00000016 H'00000015 H'00000030
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 2, r.sigId: 27845100 gsn: 664 "Unknown" prio: 1
s.bn: 250 "DBDICT", s.proc: 2, s.sigId: 27845098 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'00000422 H'00fa0002 H'00000423
--------------- Signal ----------------
r.bn: 245 "DBTC", r.proc: 2, r.sigId: 27845099 gsn: 342 "PACKED_SIGNAL" prio: 1
s.bn: 247 "DBLQH", s.proc: 5, s.sigId: 26560312 length: 7 trace: 0 #sec: 0 fragInf: 0
Signal data: H'400001fd H'0000807f H'00f70005 H'00000000 H'00001d40 H'00700800 H'00000000
--------- Begin Packed Signals --------
--------------- Signal ----------------
r.bn: 245 "DBTC", length: 7 "LQHKEYCONF"
Signal data: H'400001fd H'0000807f H'00f70005 H'00000000 H'00001d40 H'00700800 H'00000000
--------- End Packed Signals ----------
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 2, r.sigId: 27845098 gsn: 730 "DROP_OBJ_CONF" prio: 1
s.bn: 250 "DBDICT", s.proc: 5, s.sigId: 26560310 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'00fa0005 H'00000422 H'0000000c
--------------- Signal ----------------
***EOM***

Time: Tuesday 28 Mars 2006 - 03:36:01
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: pgman.cpp
Error object: PGMAN (Line: 1601) 0x0000000a
Program: /home/ndbdev/jmiller/builds/libexec/ndbd
Pid: 22563
Trace: /space/run/ndb_3_trace.log.1
Version: Version 5.1.9 (beta)
--------------- Signal ----------------
r.bn: 259 "TSMAN", r.proc: 3, r.sigId: 14136141 gsn: 722 "DROP_FILE_REQ" prio: 1
s.bn: 250 "DBDICT", s.proc: 3, s.sigId: 14136140 length: 6 trace: 0 #sec: 0 fragInf: 0
 H'000002be H'00fa0003 H'00000002 H'00000016 H'00000015 H'0000001f
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 3, r.sigId: 14136140 gsn: 664 "Unknown" prio: 1
s.bn: 250 "DBDICT", s.proc: 2, s.sigId: -1 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'000002bd H'00fa0002 H'000002be
--------------- Signal ----------------
r.bn: 247 "DBLQH", r.proc: 3, r.sigId: 14136139 gsn: 4 "ATTRINFO" prio: 1
s.bn: 247 "DBLQH", s.proc: 2, s.sigId: 14855994 length: 10 trace: 1 #sec: 0 fragInf: 0
 H'00008295 H'000013b4 H'00700800 H'20202020 H'20202020 H'00020010 H'31434241
 H'20202020 H'20202020 H'20202020
--------------- Signal ----------------
r.bn: 247 "DBLQH", r.proc: 3, r.sigId: 14136138 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 247 "DBLQH", s.proc: 2, s.sigId: 14855994 length: 22 trace: 1 #sec: 0 fragInf: 0
 ClientPtr = H'00000083 hashValue = H'fc6836d7 tcBlockRef = H'00f50002
 transId1 = H'000013b4 transId2 = H'00700800 savePointId = H'000008a5
 Op: 2 Lock: 0 Flags: Rowid ScanInfo/noFiredTriggers: H'0
 AttrLen: 12 (5 in this) KeyLen: 1 TableId: 25 SchemaVer: 1
 FragId: 0 ReplicaNo: 1 LastReplica: 1 NextNodeId: 0
 ApiRef: H'80070008 ApiOpRef: H'000022b8 TcOpRec: H'00008295
 KeyInfo: H'0000011a
 Rowid: [ page: 0 idx: 1078 ]
 AttrInfo: H'00000004 H'0000011a H'00010010 H'315a5958 H'20202020
--------------- Signal ----------------
***EOM***

Time: Tuesday 28 Mars 2006 - 03:35:08
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: pgman.cpp
Error object: PGMAN (Line: 1601) 0x0000000a
Program: /home/ndbdev/jmiller/builds/libexec/ndbd
Pid: 25084
Trace: /space/run/ndb_4_trace.log.1
Version: Version 5.1.9 (beta)
--------------- Signal ----------------
r.bn: 259 "TSMAN", r.proc: 4, r.sigId: 13162662 gsn: 722 "DROP_FILE_REQ" prio: 1

s.bn: 250 "DBDICT", s.proc: 4, s.sigId: 13162661 length: 6 trace: 0 #sec: 0 fragInf: 0
 H'000002be H'00fa0004 H'00000002 H'00000016 H'00000015 H'0000001f
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 4, r.sigId: 13162661 gsn: 664 "Unknown" prio: 1
s.bn: 250 "DBDICT", s.proc: 2, s.sigId: -1 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'000002bd H'00fa0002 H'000002be
--------------- Signal ----------------
r.bn: 245 "DBTC", r.proc: 4, r.sigId: 13162660 gsn: 409 "TIME_SIGNAL" prio: 1
s.bn: 252 "QMGR", s.proc: 4, s.sigId: 13162656 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000004
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 4, r.sigId: 13162659 gsn: 164 "CONTINUEB" prio: 1
s.bn: 253 "NDBFS", s.proc: 4, s.sigId: 13162655 length: 1 trace: 0 #sec: 0 fragInf: 0
 Scanning the memory channel again with no delay
--------------- Signal ----------------
r.bn: 250 "DBDICT", r.proc: 4, r.sigId: 13162658 gsn: 255 "FSCLOSECONF" prio: 1
s.bn: 253 "NDBFS", s.proc: 4, s.sigId: 13162655 length: 1 trace: 0 #sec: 0 fragInf: 0
 UserPointer: 0
--------------- Signal ----------------
***EOM***

Time: Tuesday 28 Mars 2006 - 03:49:26
Status: Temporary error, restart node
Message: Arbitrator shutdown, please investigate error(s) on other node(s) (Arbitration error)
Error: 2305
Error data: Arbitrator decided to shutdown this node
Error object: QMGR (Line: 3899) 0x0000000a
Program: /home/ndbdev/jmiller/builds/libexec/ndbd
Pid: 24722
Trace: /space/run/ndb_5_trace.log.1
Version: Version 5.1.9 (beta)
--------------- Signal ----------------
r.bn: 252 "QMGR", r.proc: 5, r.sigId: 26565127 gsn: 343 "PREP_FAILCONF" prio: 0
s.bn: 252 "QMGR", s.proc: 5, s.sigId: 26565126 length: 2 trace: 8 #sec: 0 fragInf: 0
 H'00000005 H'00000004
--------------- Signal ----------------
r.bn: 252 "QMGR", r.proc: 5, r.sigId: 26565126 gsn: 126 "CLOSE_COMCONF" prio: 0
s.bn: 254 "CMVMI", s.proc: 5, s.sigId: 26565125 length: 19 trace: 8 #sec: 0 fragInf: 0
 xxxBlockRef = (252, 5) failNo = 4 noOfNodes = 1
 Nodes:  2
--------------- Signal ----------------
r.bn: 254 "CMVMI", r.proc: 5, r.sigId: 26565125 gsn: 127 "CLOSE_COMREQ" prio: 0
s.bn: 252 "QMGR", s.proc: 5, s.sigId: 26565124 length: 5 trace: 8 #sec: 0 fragInf: 0
 xxxBlockRef = (252, 5) failNo = 4 noOfNodes = 1
 Nodes:  2
--------------- Signal ----------------
r.bn: 252 "QMGR", r.proc: 5, r.sigId: 26565124 gsn: 345 "PREP_FAILREQ" prio: 0
s.bn: 252 "QMGR", s.proc: 5, s.sigId: 26565123 length: 5 trace: 8 #sec: 0 fragInf: 0
 xxxBlockRef = (252, 5) failNo = 4 noOfNodes = 1
 Nodes:  2

***EOM***

Create Insert Drop for Disk Data

Attachment: cid_ndb_dd2.pl (application/x-perl, text), 6.98 KiB.

Looper Shell

Attachment: looper.sh (application/x-shellscript, text), 434 bytes.

Data file for looper

Attachment: table1.dat (application/octet-stream, text), 960 bytes.

Data file for looper

Attachment: table.dat (application/octet-stream, text), 153 bytes.

mysqld ID 2
060328  2:59:20 [ERROR] NDB: Could not get apply status share
060328  3:00:41 [ERROR] NDB: CREATE TABLE IF NOT EXISTS cluster.schema ( db VARBINARY(63) NOT NULL, name VARBINARY(63) NOT NULL, slock BINARY(32) NOT NULL, query BLOB NOT NULL, node_id INT UNSIGNED NOT NULL, epoch BIGINT UNSIGNED NOT NULL, id INT UNSIGNED NOT NULL, version INT UNSIGNED NOT NULL, type INT UNSIGNED NOT NULL, PRIMARY KEY USING HASH (db,name) ) ENGINE=NDB: error Can't create table 'cluster.schema' (errno: 701) 1005 1 1
060328  3:00:42 [Note] NDB: missing frm for cluster.schema, discovering...
060328  3:00:43 [Note] NDB: missing frm for cluster.apply_status, discovering...
060328  3:16:00 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:16:23 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
Error: NdbObjectIdMap::::unmap(0, 0x90f4e08) obj=0x8f6dfe8
060328  3:16:48 [ERROR] NDB Binlog: Creating NdbEventOperation failed for REPL$TESTER2/t1
060328  3:16:48 [ERROR] NDB Binlog:FAILED CREATE (DISCOVER) EVENT OPERATIONS Event: REPL$TESTER2/t1
060328  3:19:16 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:19:38 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:20:52 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:21:15 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:22:31 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:22:54 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:29:01 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:29:24 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:33:03 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:33:28 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:39:44 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:40:06 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:40:32 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:40:54 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:42:59 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:43:21 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:44:59 [ERROR] NDB: CREATE DATABASE TESTER2: error Can't create database 'TESTER2'; database exists 1007 1 1
060328  3:46:14 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:46:36 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
out of order bucket: 2 gci: 1758 m_latestGCI: 1754
out of order bucket: 2 gci: 1758 m_latestGCI: 1754

MySQLD ID 1
060328  2:59:19 [ERROR] NDB: Could not get apply status share
060328  3:16:00 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:16:23 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388600
read_buffer_size=131072
max_used_connections=2
max_connections=100
threads_connected=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225791 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x9c41188
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xb7f39924, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81cfe00
0xd21420
0x835f4fa
0x835f4fa
0x835b546
0x8362150
0x835e267
0x8334fc6
0x8335882
0x832cdbd
0x82a1d1c
0x8323418
0x83337ef
0x8336b42
0x9fdb80
0x9559ce
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/Using_stack_trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at (nil)  is invalid pointer
thd->thread_id=1
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.

Number of processes running now: 0
060328 03:17:37  mysqld restarted
060328  3:17:37 [Note] Starting MySQL Cluster Binlog Thread
060328  3:17:38 [Note] /home/ndbdev/jmiller/builds/libexec/mysqld: ready for connections.
Version: '5.1.9-beta-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution
060328  3:19:16 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:19:38 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:20:52 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:21:15 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:22:31 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:22:54 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:29:01 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:29:24 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:33:03 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:33:28 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:39:44 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:40:06 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:40:32 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:40:54 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:42:59 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:43:21 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:44:59 [ERROR] NDB: CREATE DATABASE TESTER2: error Can't create database 'TESTER2'; database exists 1007 1 1
060328  3:46:14 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:46:36 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
out of order bucket: 2 gci: 1758 m_latestGCI: 1754
out of order bucket: 2 gci: 1758 m_latestGCI: 1754

MySQLD ID 3
060328  2:52:14 [Note] Starting MySQL Cluster Binlog Thread
060328  2:52:14 [Note] /home/ndbdev/jmiller/builds/libexec/mysqld: ready for connections.
Version: '5.1.9-beta-log'  socket: '/tmp/mysql2.sock'  port: 3307  Source distribution
060328  2:59:20 [ERROR] NDB: Could not get apply status share
060328  3:00:41 [ERROR] NDB: CREATE TABLE IF NOT EXISTS cluster.schema ( db VARBINARY(63) NOT NULL, name VARBINARY(63) NOT NULL, slock BINARY(32) NOT NULL, query BLOB NOT NULL, node_id INT UNSIGNED NOT NULL, epoch BIGINT UNSIGNED NOT NULL, id INT UNSIGNED NOT NULL, version INT UNSIGNED NOT NULL, type INT UNSIGNED NOT NULL, PRIMARY KEY USING HASH (db,name) ) ENGINE=NDB: error Can't create table 'cluster.schema' (errno: 701) 1005 1 1
060328  3:00:42 [Note] NDB: missing frm for cluster.schema, discovering...
060328  3:00:42 [ERROR] NDB: CREATE TABLE IF NOT EXISTS cluster.apply_status ( server_id INT UNSIGNED NOT NULL, epoch BIGINT UNSIGNED NOT NULL,  PRIMARY KEY USING HASH (server_id) ) ENGINE=NDB: error Can't create table 'cluster.apply_status' (errno: 156) 1005 1 1
060328  3:00:43 [Note] NDB: missing frm for cluster.apply_status, discovering...
out of order bucket: 2 gci: 1758 m_latestGCI: 1754
out of order bucket: 2 gci: 1758 m_latestGCI: 1754

MySQLD ID 4: Thousand of lines of: 
060328 16:15:12 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'
060328 16:15:12 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'
060328 16:15:12 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'
060328 16:15:13 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'
060328 16:15:13 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'
060328 16:15:13 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'
060328 16:15:13 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'
060328 16:15:13 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'
060328 16:15:13 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'
060328 16:15:13 [ERROR] /home/ndbdev/jmiller/builds/libexec/mysqld: Incorrect information in file: './hlr/matable.frm'

MySQLD ID 5

060328  2:59:19 [ERROR] NDB: Could not get apply status share
060328  3:00:41 [ERROR] NDB: CREATE TABLE IF NOT EXISTS cluster.schema ( db VARBINARY(63) NOT NULL, name VARBINARY(63) NOT NULL, slock BINARY(32) NOT NULL, query BLOB NOT NULL, node_id INT UNSIGNED NOT NULL, epoch BIGINT UNSIGNED NOT NULL, id INT UNSIGNED NOT NULL, version INT UNSIGNED NOT NULL, type INT UNSIGNED NOT NULL, PRIMARY KEY USING HASH (db,name) ) ENGINE=NDB: error Can't create table 'cluster.schema' (errno: 701) 1005 1 1
060328  3:00:42 [Note] NDB: missing frm for cluster.schema, discovering...
060328  3:00:42 [ERROR] NDB: CREATE TABLE IF NOT EXISTS cluster.apply_status ( server_id INT UNSIGNED NOT NULL, epoch BIGINT UNSIGNED NOT NULL,  PRIMARY KEY USING HASH (server_id) ) ENGINE=NDB: error Can't create table 'cluster.apply_status' (errno: 701) 1005 1 1
060328  3:00:43 [Note] NDB: missing frm for cluster.apply_status, discovering...
060328  3:16:00 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:16:23 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:19:16 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:19:38 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:20:52 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:21:15 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:22:31 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:22:54 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:29:01 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:29:24 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:33:03 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:33:28 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:39:44 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:40:06 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:40:32 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:40:54 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:42:59 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:43:21 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
060328  3:44:59 [ERROR] NDB: CREATE DATABASE TESTER2: error Can't create database 'TESTER2'; database exists 1007 1 1
060328  3:46:14 [ERROR] Could not discover table 'TESTER2.t1' from binlog schema event 'CREATE TABLE t1 (c1 INT, c2 CHAR(4), c3 FLOAT,
                         c4 DOUBLE, c5 BIT(4), c6 VARCHAR(100), PRIMARY KEY (c1))
                         TABLESPACE TESTER2_TS STORAGE DISK ENGINE=NDB' from node 9
060328  3:46:36 [ERROR] NDB: DROP DATABASE TESTER2: error Can't drop database 'TESTER2'; database doesn't exist 1008 1 1
out of order bucket: 2 gci: 1758 m_latestGCI: 1754
out of order bucket: 2 gci: 1758 m_latestGCI: 1754

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/5297

pushed to 5.1.10

Parallell create/drop table and create/drop logfile group/tablespace/datafile/undofile
  could lead to node failure (and this to all sorts of subsequent problems...)

Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html

Additional info:

Documented in 5.1.10 changelog; closed.