Bug #43513 NDBMTD failed ndbrequire in SimulatedBlock.cpp DBTC (Line: 501) 0x00000006
Submitted: 9 Mar 2009 21:02 Modified: 10 Apr 2009 21:41
Reporter: Jonathan Miller Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql-5.1-telco-6.4 OS:Linux
Assigned to: Jonathan Miller CPU Architecture:Any

[9 Mar 2009 21:02] Jonathan Miller 
Description:
ACRT (4-dn-mt-8-mixed)

4 Data Node
Mixed NDBD/NDBMTD configuration 
8 Execution threads (for the NDBMTD's spawned)

ACRT produced the crash/error message listed below while running sysbench

Time: Saturday 7 March 2009 - 22:14:54
Status: Temporary error, restart node
Message: Internal program error (failed ndbrequire) (Internal error, programming error or missing error message, please report a bug)
Error: 2341
Error data: SimulatedBlock.cpp
Error object: DBTC (Line: 501) 0x00000006
Program: /data0/cr_autotest/libexec/ndbmtd
Pid: 13535
Trace: ./ndb_3_trace.log.1 ./ndb_3_trace.log.1_t1 ./ndb_3_trace.log.1_t2 ./ndb_3_trace.log.1_t3 ./ndb_3_trace.log.1_t4 ./ndb_3_trace.log.1_t5 ./ndb_

How to repeat:
ACRT/sysbench/4-dn-mt-8-mixed on mysql-5.1-telco-6.4 

(/data0/cr_autotest/bin/sysbench --num-threads=4 --max-requests=0 --max-time=60 --test=oltp --oltp-table-size=100000  --oltp-point-selects=1 --oltp-simple-ranges=1 --oltp-sum-ranges=1 --oltp-order-ranges=0 --oltp-dis)
[9 Mar 2009 21:02] Jonathan Miller 
ndb_3_trace.log.1_t1

SUMA    004510 004536 003785 005505 
LGMAN   001888 
LGMAN   001888 
PGMAN   000239 000252 001080 

--------------- Signal ----------------
r.bn: 247 "DBLQH", r.proc: 3, r.sigId: 343212 gsn: 761 "STOP_FOR_CRASH" prio: 0
s.bn: 0 "SYS", s.proc: 0, s.sigId: 0 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000000
--------------- Signal ----------------
r.bn: 261/5 "PGMAN", r.proc: 3, r.sigId: 343211 gsn: 164 "CONTINUEB" prio: 0
s.bn: 261/5 "PGMAN", s.proc: 3, s.sigId: 343188 length: 1 trace: 1 #sec: 0 fragInf: 0
 H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 3, r.sigId: 343210 gsn: 699 "SUB_GCP_COMPLETE_ACK" prio: 1
s.bn: 257 "SUMA", s.proc: 2, s.sigId: -1 length: 5 trace: 2 #sec: 0 fragInf: 0
 H'00000674 H'01010002 H'00000001 H'0000000f H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 3, r.sigId: 343209 gsn: 699 "SUB_GCP_COMPLETE_ACK" prio: 1
s.bn: 257 "SUMA", s.proc: 3, s.sigId: 343208 length: 5 trace: 2 #sec: 0 fragInf: 0
 H'00000674 H'01010003 H'00000001 H'0000000f H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 3, r.sigId: 343208 gsn: 699 "SUB_GCP_COMPLETE_ACK" prio: 0
s.bn: 3 "UUNET", s.proc: 6, s.sigId: -1 length: 5 trace: 2 #sec: 0 fragInf: 0
 H'00000674 H'0fa20006 H'00000001 H'0000000f H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 3, r.sigId: 343207 gsn: 593 "SUB_GCP_COMPLETE_REP" prio: 1
s.bn: 247/4 "DBLQH", s.proc: 3, s.sigId: 13111353 length: 5 trace: 2 #sec: 0 fragInf: 0
 gci_hi: 674 gci_lo: f
 H'00000674 H'00f60002 H'00000001 H'0000000f H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 3, r.sigId: 343206 gsn: 593 "SUB_GCP_COMPLETE_REP" prio: 1
s.bn: 247/3 "DBLQH", s.proc: 3, s.sigId: 49840986 length: 5 trace: 2 #sec: 0 fragInf: 0
 gci_hi: 674 gci_lo: f
 H'00000674 H'00f60002 H'00000001 H'0000000f H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 3, r.sigId: 343205 gsn: 593 "SUB_GCP_COMPLETE_REP" prio: 1
s.bn: 247/1 "DBLQH", s.proc: 3, s.sigId: 65352614 length: 5 trace: 2 #sec: 0 fragInf: 0
 gci_hi: 674 gci_lo: f
 H'00000674 H'00f60002 H'00000001 H'0000000f H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 3, r.sigId: 343204 gsn: 593 "SUB_GCP_COMPLETE_REP" prio: 1
s.bn: 247/2 "DBLQH", s.proc: 3, s.sigId: 15889634 length: 5 trace: 2 #sec: 0 fragInf: 0
 gci_hi: 674 gci_lo: f
 H'00000674 H'00f60002 H'00000001 H'0000000f H'00000002
--------------- Signal ----------------
r.bn: 247 "DBLQH", r.proc: 3, r.sigId: 343203 gsn: 593 "SUB_GCP_COMPLETE_REP" prio: 1
s.bn: 246 "DBDIH", s.proc: 3, s.sigId: 25238723 length: 5 trace: 2 #sec: 0 fragInf: 0
 gci_hi: 674 gci_lo: f
 H'00000674 H'00f60002 H'00000001 H'0000000f H'00000002
--------------- Signal ----------------
r.bn: 257 "SUMA", r.proc: 3, r.sigId: 343202 gsn: 278 "GCP_PREPARE" prio: 1
s.bn: 246 "DBDIH", s.proc: 3, s.sigId: 25238691 length: 3 trace: 2 #sec: 0 fragInf: 0
 H'00000002 H'00000674 H'00000010
[9 Mar 2009 21:02] Jonathan Miller 
ndb_3_trace.log.1_t2

BACKUP  000189 
DBLQH   002801 
DBTUP   010029 
DBLQH   002801 
DBTUP   010029 
DBLQH   002801 
DBTUP   010029 

--------------- Signal ----------------
r.bn: 247/1 "DBLQH", r.proc: 3, r.sigId: 65352619 gsn: 761 "STOP_FOR_CRASH" prio: 0
s.bn: 0 "SYS", s.proc: 0, s.sigId: 0 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000000
--------------- Signal ----------------
r.bn: 244/1 "BACKUP", r.proc: 3, r.sigId: 65352618 gsn: 164 "CONTINUEB" prio: 0
s.bn: 244/1 "BACKUP", s.proc: 3, s.sigId: 65352616 length: 1 trace: 1 #sec: 0 fragInf: 0
 H'00000006
--------------- Signal ----------------
r.bn: 261/1 "PGMAN", r.proc: 3, r.sigId: 65352617 gsn: 164 "CONTINUEB" prio: 0
s.bn: 261/1 "PGMAN", s.proc: 3, s.sigId: 65352611 length: 1 trace: 1 #sec: 0 fragInf: 0
 H'00000002
--------------- Signal ----------------
r.bn: 244/1 "BACKUP", r.proc: 3, r.sigId: 65352616 gsn: 164 "CONTINUEB" prio: 0
s.bn: 244/1 "BACKUP", s.proc: 3, s.sigId: 65352613 length: 1 trace: 1 #sec: 0 fragInf: 0
 H'00000006
--------------- Signal ----------------
r.bn: 248/1 "DBACC", r.proc: 3, r.sigId: 65352615 gsn: 164 "CONTINUEB" prio: 0
s.bn: 248/1 "DBACC", s.proc: 3, s.sigId: 65352572 length: 2 trace: 0 #sec: 0 fragInf: 0
 H'00000008 H'00000091
--------------- Signal ----------------
r.bn: 247/1 "DBLQH", r.proc: 3, r.sigId: 65352614 gsn: 593 "SUB_GCP_COMPLETE_REP" prio: 1
s.bn: 247 "DBLQH", s.proc: 3, s.sigId: 343203 length: 5 trace: 2 #sec: 0 fragInf: 0
 gci_hi: 674 gci_lo: f
 H'00000674 H'00f60002 H'00000001 H'0000000f H'00000002
[9 Mar 2009 21:02] Jonathan Miller 
ndb_3_trace.log.1_t3

DBLQH   002801 
DBTUP   010029 
DBLQH   002801 
DBTUP   010029 

--------------- Signal ----------------
r.bn: 247/2 "DBLQH", r.proc: 3, r.sigId: 15889637 gsn: 761 "STOP_FOR_CRASH" prio: 0
s.bn: 0 "SYS", s.proc: 0, s.sigId: 0 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000000
--------------- Signal ----------------
r.bn: 244/2 "BACKUP", r.proc: 3, r.sigId: 15889636 gsn: 164 "CONTINUEB" prio: 0
s.bn: 244/2 "BACKUP", s.proc: 3, s.sigId: 15889633 length: 1 trace: 1 #sec: 0 fragInf: 0
 H'00000006
--------------- Signal ----------------
r.bn: 261/2 "PGMAN", r.proc: 3, r.sigId: 15889635 gsn: 164 "CONTINUEB" prio: 0
s.bn: 261/2 "PGMAN", s.proc: 3, s.sigId: 15889630 length: 1 trace: 1 #sec: 0 fragInf: 0
 H'00000002
--------------- Signal ----------------
r.bn: 247/2 "DBLQH", r.proc: 3, r.sigId: 15889634 gsn: 593 "SUB_GCP_COMPLETE_REP" prio: 1
s.bn: 247 "DBLQH", s.proc: 3, s.sigId: 343203 length: 5 trace: 2 #sec: 0 fragInf: 0
 gci_hi: 674 gci_lo: f
 H'00000674 H'00f60002 H'00000001 H'0000000f H'00000002
[9 Mar 2009 21:03] Jonathan Miller 
ndb_3_trace.log.1_t4

DBLQH   002801 
DBTUP   010029 
DBLQH   002801 
DBTUP   010029 
DBLQH   002801 
DBTUP   010029 

--------------- Signal ----------------
r.bn: 247/3 "DBLQH", r.proc: 3, r.sigId: 49855830 gsn: 761 "STOP_FOR_CRASH" prio: 0
s.bn: 0 "SYS", s.proc: 0, s.sigId: 0 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000000
--------------- Signal ----------------
r.bn: 244/3 "BACKUP", r.proc: 3, r.sigId: 49855829 gsn: 164 "CONTINUEB" prio: 0
s.bn: 244/3 "BACKUP", s.proc: 3, s.sigId: 49841299 length: 3 trace: 4 #sec: 0 fragInf: 0
 H'00000000 H'00000000 H'0000163f
--------------- Signal ----------------
r.bn: 244/3 "BACKUP", r.proc: 3, r.sigId: 49855828 gsn: 164 "CONTINUEB" prio: 0
s.bn: 244/3 "BACKUP", s.proc: 3, s.sigId: 49852585 length: 2 trace: 4 #sec: 0 fragInf: 0
 H'00000002 H'00000000
--------------- Signal ----------------
r.bn: 244/3 "BACKUP", r.proc: 3, r.sigId: 49855827 gsn: 164 "CONTINUEB" prio: 0
s.bn: 244/3 "BACKUP", s.proc: 3, s.sigId: 49840621 length: 1 trace: 1 #sec: 0 fragInf: 0
 H'00000006
--------------- Signal ----------------
r.bn: 248/3 "DBACC", r.proc: 3, r.sigId: 49855826 gsn: 253 "EXPANDCHECK2" prio: 1
s.bn: 248/3 "DBACC", s.proc: 3, s.sigId: 49855825 length: 3 trace: 1 #sec: 0 fragInf: 0
 H'00000009 H'000001e5 H'000003ff
[9 Mar 2009 21:03] Jonathan Miller 
ndb_3_trace.log.1_t5

BACKUP  000189 000300 004677 
DBLQH   002801 
DBTUP   010029 
DBLQH   002801 
DBTUP   010029 

--------------- Signal ----------------
r.bn: 247/4 "DBLQH", r.proc: 3, r.sigId: 13111375 gsn: 761 "STOP_FOR_CRASH" prio: 0
s.bn: 0 "SYS", s.proc: 0, s.sigId: 0 length: 1 trace: 0 #sec: 0 fragInf: 0
 H'00000000
--------------- Signal ----------------
r.bn: 244/4 "BACKUP", r.proc: 3, r.sigId: 13111374 gsn: 164 "CONTINUEB" prio: 0
s.bn: 244/4 "BACKUP", s.proc: 3, s.sigId: 13111356 length: 3 trace: 4 #sec: 0 fragInf: 0
 H'00000000 H'00000000 H'0000163f
--------------- Signal ----------------
r.bn: 244/4 "BACKUP", r.proc: 3, r.sigId: 13111373 gsn: 164 "CONTINUEB" prio: 0
s.bn: 244/4 "BACKUP", s.proc: 3, s.sigId: 13111352 length: 1 trace: 1 #sec: 0 fragInf: 0
 H'00000006
--------------- Signal ----------------
r.bn: 261/4 "PGMAN", r.proc: 3, r.sigId: 13111372 gsn: 164 "CONTINUEB" prio: 0
s.bn: 261/4 "PGMAN", s.proc: 3, s.sigId: 13111342 length: 1 trace: 1 #sec: 0 fragInf: 0
 H'00000002
--------------- Signal ----------------
r.bn: 244/4 "BACKUP", r.proc: 3, r.sigId: 13111371 gsn: 351 "SCAN_FRAGCONF" prio: 1
s.bn: 247/4 "DBLQH", s.proc: 3, s.sigId: 13111370 length: 6 trace: 4 #sec: 0 fragInf: 0
 H'00000000 H'00000000 H'00000002 H'00000000 H'0f400300 H'00000000
[9 Mar 2009 21:03] Jonathan Miller 
rundate_2009-03-07_18:55
revno: 2918
revision-id: msvensson@mysql.com-20090306160957-nwj1ulv16ozti2ij
parent: jon@mysql.com-20090306095001-u7mxm4do37avp89h
parent: msvensson@mysql.com-20090306155054-sfmpvio2hxaquyey
committer: Magnus Svensson 
branch nick: mysql-5.1-telco-6.4
timestamp: Fri 2009-03-06 17:09:57 +0100
message:
  Merge
[10 Apr 2009 21:41] Jonathan Miller 
ACRT has not reproduced this failure.

Failure:
2009-03-07 18:55 Tests: 0/1 sysbench-oltp-tester.sh

Test was rerun and passed:
2009-03-25 23:17 Tests: 1/1