Bug #41398 NDBMT MaxNoOfExecutionThreads=6 failure in dbtup/DbtupDiskAlloc.cpp
Submitted: 11 Dec 2008 14:55 Modified: 12 Mar 2009 18:15
Reporter: Jonathan Miller Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Disk Data Severity:S2 (Serious)
Version:mysql-5.1-telco-6.4 OS:Linux
Assigned to: Pekka Nousiainen CPU Architecture:Any

[11 Dec 2008 14:55] Jonathan Miller 
Description:
Time: Thursday 11 December 2008 - 13:56:20
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtup/DbtupDiskAlloc.cpp
Error object: DBTUP (Line: 582) 0x00000006
Program: /data0/cr_autotest/libexec/ndbmtd
Pid: 3376
Trace: ./ndb_3_trace.log.1 ./ndb_3_trace.log.1_t1 ./ndb_3_trace.log.1_t2 ./ndb_3_trace.log.1_t3 ./ndb_3_trace.log.1_t4
Version: mysql-5.1. 

How to repeat:
ACRT 2-dn-mt-6 test using  ndbatomics-dd-tester.sh
[22 Dec 2008 21:45] Jonathan Miller 
result-4-dn-mt-8-mysql-5.1-telco-6.4/2008-12-22_20:34

Time: Tuesday 23 December 2008 - 00:29:53
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtup/DbtupDiskAlloc.cpp
Error object: DBTUP (Line: 582) 0x00000006
Program: /data0/cr_autotest/libexec/ndbmtd
Pid: 337
Trace: ./ndb_4_trace.log.1 ./ndb_4_trace.log.1_t1 ./ndb_4_trace.log.1_t2 ./ndb_4_trace.log.1_t3 ./ndb_4_trace.log.1_t4 ./ndb_4_trace.log.1_t
[7 Jan 2009 9:54] Pekka Nousiainen 
line numbers can change.
context for line 582:
Dbtup::disk_page_prealloc(...
  if (newPageBits != (Uint32)pageBits)...
    ddassert(ext.p->m_free_page_count[pageBits] > 0);
[7 Jan 2009 21:27] Jonathan Miller 
Current byte-offset of file-pointer is: 568                       

Time: Wednesday 7 January 2009 - 13:35:31
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtup/DbtupDiskAlloc.cpp
Error object: DBTUP (Line: 582) 0x00000006
Program: /data0/cr_autotest/libexec/ndbmtd
Pid: 23313
Trace: ./ndb_5_trace.log.1 ./ndb_5_trace.log.1_t1 ./ndb_5_trace.log.1_t2 ./ndb_5_trace.log.1_t3 ./ndb_5_trace.log.1_t4
Version: mysql-5.1.
[7 Jan 2009 21:27] Jonathan Miller 
ndb_5_signal.log

3COMMIT:  activeCreat: 0INSERT key: 4294965260
INSERT tab: 2 frag: 3 activeCreat: 0 key: 4294965259
 tab: 2 frag: 5 activeCreat: 0 rowid: [ m_page_no: 0 m_file_no: 62451 m_page_idx: 1841]INSERT key: 0
 tab: INSERT tab: 22 frag: 1 activeCreat: 0 rowid: [ m_page_no: 0 m_file_no: 62451 m_page_idx: 1904] key: 4294965255
 frag: 7 activeCreat: 0 key: 4294965257
INSERT tab: 2 frag: 7 activeCreat: 0 key: 4294965256
INSERT tab: 2 frag: 5 activeCreat: 0 rowid: [ m_page_no: 0 m_file_no: 62451 m_page_idx: 1848] key: 4294965263
COMMIT: INSERT tab: 2 frag: 7 activeCreat: 0 key: COMMIT: 0
INSERT tab: COMMIT: 2 frag: 3 activeCreat: 0 key: 0
COMMIT: INSERT tab: 2 frag: 3 activeCreat: 0 key: INSERT0 tab: 2 frag: 
7 activeCreat: 0 key: 0
COMMIT: INSERT tab: 2 frag: 1 activeCreat: 0 rowid: [ m_page_no: 0 m_file_no: 62451 m_page_idx: 1904] key: 0
COMMIT: INSERT tab: 2 frag: 5 activeCreat: 0 rowid: [ m_page_no: 0 m_file_no: 62451 m_page_idx: 1848] key: 0
[7 Jan 2009 21:29] Jonathan Miller 
ndb_5_trace.log.1

DBTC    004430 
DBDIH   009346 
DBTC    004430 
DBTC    004430 

--------------- Signal ----------------
r.bn: 251 "NDBCNTR", r.proc: 5, r.sigId: 52602659 gsn: 761 "STOP_FOR_CRASH" prio: 0
s.bn: 0 "SYS", s.proc: 0, s.sigId: 0 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000000
--------------- Signal ----------------
r.bn: 246 "DBDIH", r.proc: 5, r.sigId: 52602658 gsn: 593 "SUB_GCP_COMPLETE_REP" prio: 0
s.bn: 246 "DBDIH", s.proc: 2, s.sigId: 52188509 length: 5 trace: 1 #sec: 0 fragInf: 0
 gci_hi: 9d9 gci_lo: b
 H'000009d9 H'00f60002 H'00000001 H'0000000b H'00000002
--------------- Signal ----------------
r.bn: 252 "QMGR", r.proc: 5, r.sigId: 52602657 gsn: 3 "API_REGREQ" prio: 0
s.bn: 4002 "API", s.proc: 1, s.sigId: 0 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'0fa20001 H'00060401 H'0005011e
--------------- Signal ----------------
r.bn: 253 "NDBFS", r.proc: 5, r.sigId: 52602656 gsn: 272 "FSWRITEREQ" prio: 0
s.bn: 247/2 "DBLQH", s.proc: 5, s.sigId: 143776090 length: 15 trace: 1 #sec: 0 fragInf: 0
 UserPointer: 16
 FilePointer: 61
 UserReference: H'04f70005 Operation flag: H'00000002 (No sync, Format=List of mem pages)
 varIndex: 1
 numberOfPages: 8
 pageData:  H'00000055,  H'000000b1,  H'00000057,  H'00000006,  H'00000078,  H'0000000d,  H'00000081,  H'00000014,  H'000011e8,
[7 Jan 2009 21:30] Jonathan Miller 
ndb_5_trace.log.1_t1

SUMA    004510 004563 
SUMA    004510 004536 003785 
SUMA    004510 004536 003785 005505 
TSMAN   001937 000454 
PGMAN   001600 

--------------- Signal ----------------
r.bn: 247 "DBLQH", r.proc: 5, r.sigId: 410820 gsn: 761 "STOP_FOR_CRASH" prio: 0
s.bn: 0 "SYS", s.proc: 0, s.sigId: 0 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000000
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 5, r.sigId: 410819 gsn: 699 "SUB_GCP_COMPLETE_ACK" prio: 1
s.bn: 257 "SUMA", s.proc: 4, s.sigId: 427319 length: 5 trace: 1 #sec: 0 fragInf: 0
 H'000009d9 H'01010004 H'00000001 H'0000000b H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 5, r.sigId: 410818 gsn: 699 "SUB_GCP_COMPLETE_ACK" prio: 1
s.bn: 257 "SUMA", s.proc: 5, s.sigId: 410817 length: 5 trace: 1 #sec: 0 fragInf: 0
 H'000009d9 H'01010005 H'00000001 H'0000000b H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 5, r.sigId: 410817 gsn: 699 "SUB_GCP_COMPLETE_ACK" prio: 0
s.bn: 5 "UUNET", s.proc: 6, s.sigId: -1 length: 5 trace: 1 #sec: 0 fragInf: 0
 H'000009d9 H'0fa20006 H'00000001 H'0000000b H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 5, r.sigId: 410816 gsn: 593 "SUB_GCP_COMPLETE_REP" prio: 1
s.bn: 247/2 "DBLQH", s.proc: 5, s.sigId: 143776446 length: 5 trace: 1 #sec: 0 fragInf: 0
 gci_hi: 9d9 gci_lo: b
 H'000009d9 H'00f60002 H'00000001 H'0000000b H'00000002
--------------- Signal ----------------
[7 Jan 2009 21:31] Jonathan Miller 
/ndb_5_trace.log.1_t2

DBTUP   000497 001335 016603 016747 016603 016747 016603 016747 
        016603 016747 016603 017735 017680 016603 017735 017680 
        016603 017680 016603 017735 017680 016603 017735 017680 
        016603 017833 017777 017779 001470 025100 
DBACC   002322 

--------------- Signal ----------------
r.bn: 247/3 "DBLQH", r.proc: 5, r.sigId: 143004100 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 63412401 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003df18 hashValue = H'7b699e8b tcBlockRef = H'00f50003
 transId1 = H'0003d562 transId2 = H'00700800 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'400
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 14 SchemaVer: 9
 FragId: 3 ReplicaNo: 0 LastReplica: 1 NextNodeId: 4
 ApiRef: H'80070008 ApiOpRef: H'0000017c
 AttrInfo: 
--------------- Signal ----------------
r.bn: 247/3 "DBLQH", r.proc: 5, r.sigId: 143004099 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 63412396 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003d8e8 hashValue = H'c38c7de8 tcBlockRef = H'00f50003
 transId1 = H'0003d562 transId2 = H'00700800 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'400
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 14 SchemaVer: 9
 FragId: 3 ReplicaNo: 0 LastReplica: 1 NextNodeId: 4
 ApiRef: H'80070008 ApiOpRef: H'00000168
 AttrInfo: 
--------------- Signal ----------------
r.bn: 247/3 "DBLQH", r.proc: 5, r.sigId: 143004098 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 63412377 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003d958 hashValue = H'30080cf5 tcBlockRef = H'00f50003
 transId1 = H'0003d562 transId2 = H'00700800 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'400
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 14 SchemaVer: 9
 FragId: 3 ReplicaNo: 0 LastReplica: 1 NextNodeId: 4
 ApiRef: H'80070008 ApiOpRef: H'0000011c
 AttrInfo: 
--------------- Signal ----------------
r.bn: 247/3 "DBLQH", r.proc: 5, r.sigId: 143004097 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 63412373 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003db03 hashValue = H'07e10199 tcBlockRef = H'00f50003
 transId1 = H'0003d562 transId2 = H'00700800 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'400
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 14 SchemaVer: 9
 FragId: 3 ReplicaNo: 0 LastReplica: 1 NextNodeId: 4
 ApiRef: H'80070008 ApiOpRef: H'0000010c
 AttrInfo:
[7 Jan 2009 21:31] Jonathan Miller 
ndb_5_trace.log.1_t3

DBLQH   005047 000342 
DBTUP   000497 001335 016603 016747 016603 016747 016603 016747 
        016603 016747 016603 017735 017680 016603 017735 017680 
        016603 017680 016603 017735 017680 016603 017735 017680 
        016603 017833 017777 017779 001470 025100 
DBACC   002322 090432 
DBTUP   090436 090581 000582 

--------------- Signal ----------------
r.bn: 247/4 "DBLQH", r.proc: 5, r.sigId: 143776702 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 63412511 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003de33 hashValue = H'e047d90b tcBlockRef = H'00f50003
 transId1 = H'0003d562 transId2 = H'00700800 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'400
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 14 SchemaVer: 9
 FragId: 7 ReplicaNo: 0 LastReplica: 1 NextNodeId: 4
 ApiRef: H'80070008 ApiOpRef: H'0000032c
 AttrInfo: 
--------------- Signal ----------------
r.bn: 247/4 "DBLQH", r.proc: 5, r.sigId: 143776701 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 63412507 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003e046 hashValue = H'4ea9a699 tcBlockRef = H'00f50003
 transId1 = H'0003d562 transId2 = H'00700800 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'400
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 14 SchemaVer: 9
 FragId: 7 ReplicaNo: 0 LastReplica: 1 NextNodeId: 4
 ApiRef: H'80070008 ApiOpRef: H'0000031c
 AttrInfo: 
--------------- Signal ----------------
r.bn: 247/4 "DBLQH", r.proc: 5, r.sigId: 143776700 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 63412481 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003d998 hashValue = H'5741dcb2 tcBlockRef = H'00f50003
 transId1 = H'0003d562 transId2 = H'00700800 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'400
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 14 SchemaVer: 9
 FragId: 7 ReplicaNo: 0 LastReplica: 1 NextNodeId: 4
 ApiRef: H'80070008 ApiOpRef: H'000002b4
 AttrInfo: 
--------------- Signal ----------------
r.bn: 247/4 "DBLQH", r.proc: 5, r.sigId: 143776699 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 63412478 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003e425 hashValue = H'd142ae22 tcBlockRef = H'00f50003
 transId1 = H'0003d562 transId2 = H'00700800 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'400
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 14 SchemaVer: 9
 FragId: 7 ReplicaNo: 0 LastReplica: 1 NextNodeId: 4
 ApiRef: H'80070008 ApiOpRef: H'000002a8
 AttrInfo:
[7 Jan 2009 21:32] Jonathan Miller 
ndb_5_trace.log.1_t4

CMVMI   000238 
CMVMI   000238 
CMVMI   000238 
CMVMI   000238 
CMVMI   000238 

--------------- Signal ----------------
r.bn: 254 "CMVMI", r.proc: 5, r.sigId: 13238 gsn: 761 "STOP_FOR_CRASH" prio: 0
s.bn: 0 "SYS", s.proc: 0, s.sigId: 0 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000000
--------------- Signal ----------------
r.bn: 254 "CMVMI", r.proc: 5, r.sigId: 13237 gsn: 247 "EVENT_REP" prio: 1
s.bn: 0 "SYS", s.proc: 5, s.sigId: 0 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'00000028 H'00000003 H'000003c9
--------------- Signal ----------------
r.bn: 254 "CMVMI", r.proc: 5, r.sigId: 13236 gsn: 247 "EVENT_REP" prio: 0
s.bn: 0 "SYS", s.proc: 5, s.sigId: 0 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'00000029 H'00000006 H'0000002b
--------------- Signal ----------------
r.bn: 254 "CMVMI", r.proc: 5, r.sigId: 13235 gsn: 247 "EVENT_REP" prio: 0
s.bn: 0 "SYS", s.proc: 5, s.sigId: 0 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'00000029 H'00000004 H'000010db
--------------- Signal ----------------
r.bn: 254 "CMVMI", r.proc: 5, r.sigId: 13234 gsn: 247 "EVENT_REP" prio: 1
s.bn: 0 "SYS", s.proc: 5, s.sigId: 0 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'00000028 H'00000004 H'000013c0
--------------- Signal ----------------
r.bn: 254 "CMVMI", r.proc: 5, r.sigId: 13233 gsn: 247 "EVENT_REP" prio: 0
s.bn: 0 "SYS", s.proc: 5, s.sigId: 0 length: 3 trace: 0 #sec: 0 fragInf: 0
 H'00000029 H'00000003 H'00001560
[12 Jan 2009 8:56] Pekka Nousiainen 
Jeb,
to get this bug, is a system restart required?
[22 Jan 2009 21:41] Jonathan Miller 
pekka: no system restarts are done for ndbatomics-dd-tester.sh
[15 Feb 2009 0:09] Jonathan Miller 
HI Pekka,

was able to hit it again today using a blob/disk data test:

Time: Saturday 14 February 2009 - 23:53:28
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: dbtup/DbtupDiskAlloc.cpp
Error object: DBTUP (Line: 582) 0x00000006
Program: /data0/cr_autotest/libexec/ndbmtd
Pid: 18055
Trace: ./ndb_3_trace.log.1 ./ndb_3_trace.log.1_t1 ./ndb_3_trace.log.1_t2 ./ndb_3_trace.log.1_t3 ./ndb_3_trace.log.1_t4
Version: mysql-5.1

2-dn-mt-6-mysql-5.1-telco-6.4 == 2 NDBD, NDBMTD Max Threads = 6 on mysql-5.1-telco-6.4
[15 Feb 2009 0:11] Jonathan Miller 
2009-02-14 23:54:38 [MgmSrvr] INFO     -- Node 2: Local checkpoint 219 started. Keep GCI = 3414 oldest restorable GCI = 3417
2009-02-14 23:54:42 [MgmSrvr] ALERT    -- Node 2: Node 3 Disconnected
2009-02-14 23:54:42 [MgmSrvr] ALERT    -- Node 2: Network partitioning - arbitration required
2009-02-14 23:54:42 [MgmSrvr] INFO     -- Node 2: President restarts arbitration thread [state=7]
2009-02-14 23:54:42 [MgmSrvr] INFO     -- Node 2: Communication to Node 3 closed
2009-02-14 23:54:42 [MgmSrvr] ALERT    -- Node 1: Node 3 Disconnected
2009-02-14 23:54:42 [MgmSrvr] ALERT    -- Node 2: Arbitration won - positive reply from node 1
2009-02-14 23:54:43 [MgmSrvr] ALERT    -- Node 1: Node 3 Disconnected
2009-02-14 23:54:43 [MgmSrvr] ALERT    -- Node 3: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
2009-02-14 23:54:43 [MgmSrvr] INFO     -- Node 2: Started arbitrator node 1 [ticket=63020002a434607d]
2009-02-14 23:54:46 [MgmSrvr] INFO     -- Node 2: Communication to Node 3 opened
[15 Feb 2009 0:17] Jonathan Miller 
DBTUP   090436 090581 000582 

--------------- Signal ----------------
r.bn: 247/4 "DBLQH", r.proc: 3, r.sigId: 229168071 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 124497655 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003dcd8 hashValue = H'2a6d98fd tcBlockRef = H'00f50003
 transId1 = H'00145e66 transId2 = H'00700400 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'0
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 52 SchemaVer: 1
 FragId: 3 ReplicaNo: 0 LastReplica: 1 NextNodeId: 2
 ApiRef: H'80070004 ApiOpRef: H'00000008
 AttrInfo: 
--------------- Signal ----------------
r.bn: 247/4 "DBLQH", r.proc: 3, r.sigId: 229168070 gsn: 470 "REMOVE_MARKER_ORD" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 124497654 length: 2 trace: 1 #sec: 0 fragInf: 0
 H'00145e65 H'00700400
--------------- Signal ----------------
r.bn: 247/2 "DBLQH", r.proc: 3, r.sigId: 229168069 gsn: 470 "REMOVE_MARKER_ORD" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 124497654 length: 2 trace: 1 #sec: 0 fragInf: 0
 H'00145e65 H'00700400
[15 Feb 2009 0:20] Jonathan Miller 
Hi,

So it was doing inserts into a simple table when it cored:

INSERT/DELETE -> LIDD created.
INSERT/DELETE -> Creating simple Table DD
INSERT/DELETE -> Creating long_blob DD Table
INSERT/DELETE -> Tables Created!
INSERT/DELETE -> Loading simple table
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> 10,000 rows inserted
INSERT/DELETE -> insert into simple Error: 
Err#: 1297
Err-> Got temporary error 4010 'Node failure caused abort of transaction' from NDBCLUSTER
INSERT/DELETE -> Removing TS datafile39.dat..
[15 Feb 2009 0:24] Jonathan Miller 
Need to make a correction about "blob". The test hand not made it to the blob part before it cored. simple table looks like:

    LogMessage("INSERT/DELETE -> Creating simple Table DD");
    $failed = 0;
    $sth = $dbhM->prepare("CREATE TABLE simple (aid INT, bid INT,
                                                 balance DECIMAL(8,2),
                                                 filler CHAR(80),
                                                 PRIMARY KEY (aid))
                                                 TABLESPACE $tsName
                                    STORAGE DISK ENGINE=$engineType;")
      or FailedSQL("Prepare Create simple table DD Error: ");
    $sth->execute()
      or SQL_ExecuteError("Create simple DD Table Error: ",1);
    $sth->finish();
    if($failed == 1){FailedExit("Create simple DD Table Error:");}
[15 Feb 2009 0:27] Jonathan Miller 
One other thread was messing it the DBtup at the same time

DBLQH   002771 002781 
DBLQH   002771 
DBLQH   002801 
DBTUP   010029 
DBLQH   002801 
DBTUP   010029 

--------------- Signal ----------------
r.bn: 247/1 "DBLQH", r.proc: 3, r.sigId: 289180616 gsn: 761 "STOP_FOR_CRASH" prio: 0
s.bn: 0 "SYS", s.proc: 0, s.sigId: 0 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000000
--------------- Signal ----------------
r.bn: 247/3 "DBLQH", r.proc: 3, r.sigId: 289180615 gsn: 470 "REMOVE_MARKER_ORD" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 124497654 length: 2 trace: 1 #sec: 0 fragInf: 0
 H'00145e65 H'00700400
--------------- Signal ----------------
r.bn: 247/1 "DBLQH", r.proc: 3, r.sigId: 289180614 gsn: 470 "REMOVE_MARKER_ORD" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 124497654 length: 2 trace: 1 #sec: 0 fragInf: 0
 H'00145e65 H'00700400
--------------- Signal ----------------
r.bn: 247/3 "DBLQH", r.proc: 3, r.sigId: 289180613 gsn: 159 "COMPLETE" prio: 1
s.bn: 247/1 "DBLQH", s.proc: 2, s.sigId: 234383082 length: 3 trace: 1 #sec: 0 fragInf: 0
 H'00000034 H'00145e65 H'00700400
--------------- Signal ----------------
r.bn: 247/3 "DBLQH", r.proc: 3, r.sigId: 289180612 gsn: 152 "COMMIT" prio: 1
s.bn: 247/1 "DBLQH", s.proc: 2, s.sigId: 234383081 length: 5 trace: 1 #sec: 0 fragInf: 0
 H'00000034 H'00000d62 H'00145e65 H'00700400 H'00000001
--------------- Signal ----------------
r.bn: 247/3 "DBLQH", r.proc: 3, r.sigId: 289180611 gsn: 316 "LQHKEYREQ" prio: 1
s.bn: 245 "DBTC", s.proc: 3, s.sigId: 124497649 length: 13 trace: 1 #sec: 2 fragInf: 0
 ClientPtr = H'0003dcd8 hashValue = H'b2ef5c53 tcBlockRef = H'00f50003
 transId1 = H'00145e65 transId2 = H'00700400 savePointId = H'00000000
 Op: 2 Lock: 0 Flags: CommitAckMarker ScanInfo/noFiredTriggers: H'0
 AttrLen: 0 (0 in this) KeyLen: 0 TableId: 52 SchemaVer: 1
 FragId: 1 ReplicaNo: 0 LastReplica: 1 NextNodeId: 2
 ApiRef: H'80070004 ApiOpRef: H'00000008
 AttrInfo:
[27 Feb 2009 10:20] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/67820

2896 Pekka Nousiainen	2009-02-27
      bug#41398 01_pgman.diff
      pgman: take tsman mutex during LCP copy-page switch
      modified:
        storage/ndb/src/kernel/blocks/pgman.cpp
        storage/ndb/src/kernel/blocks/pgman.hpp
[27 Feb 2009 10:26] Bugs System 
Pushed into 5.1.32-ndb-6.4.4 (revid:pekka@mysql.com-20090227102105-f44543xi8rtb5lso) (version source revid:pekka@mysql.com-20090227102105-f44543xi8rtb5lso) (merge vers: 5.1.32-ndb-6.4.4) (pib:6)
[12 Mar 2009 18:15] Jon Stephens 
Documented bugfix in the NDB-7.0.4 changelog as follows:

        When using ndbmtd, repeated high-volume inserts (on the order of 10000
        rows inserted at a time) on a Disk Data table would eventually lead to
        a data node crash.