MySQL Bugs: #11667: Cluster left running overnight with no transaction failed.

Bug #11667	Cluster left running overnight with no transaction failed.
Submitted:	30 Jun 2005 13:53	Modified:	25 Jul 2005 19:06
Reporter:	Jonathan Miller	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S1 (Critical)
Version:	5.1.0-wl2325-wl1354-new	OS:	Linux (Linux)
Assigned to:	Jonas Oreland	CPU Architecture:	Any

Description:
Found that the cluster I left running overnight had failed this morning when I went to start testing. There were no transaction running on the cluster.

All files will be moved to ndb08:/space/bug#### where #### = this bug report.

Error logs:

Two of the error logs are of interest. The first seems to show maybe a null pointer.
Current byte-offset of file-pointer is: 468
Date/Time: Thursday 30 June 2005 - 04:07:34
Type of error: error
Message: No message slogan found
Fault ID: 0
Problem data:
Object of reference: QMGR (Line: 2694) 0x0000000e
ProgramName: /home/ndbdev/jmiller/builds/libexec/ndbd
ProcessID: 3386
TraceFile: ./ndb_3_trace.log.1
Version 5.1.0 (a_drop5p3)
***EOM***

The second shows GCP stop and maybe a null pointer. 
Current byte-offset of file-pointer is: 468
Date/Time: Thursday 30 June 2005 - 04:05:33
Type of error: error
Message: System error
Fault ID: 2303
Problem data: Node 2 killed this node because GCP stop was detected
Object of reference: NDBCNTR (Line: 202) 0x0000000e
ProgramName: /home/ndbdev/jmiller/builds/libexec/ndbd
ProcessID: 9515
TraceFile: ./ndb_2_trace.log.1
Version 5.1.0 (a_drop5p3)

No core files were produced.

How to repeat:
Not sure. Will let it run overnight again today and see if it can be repeated.

From the mysqld error log:
out of order bucket: 0 gci: 6968 m_latestGCI: 6964
out of order bucket: 0 gci: 6968 m_latestGCI: 6964
out of order bucket: 0 gci: 6968 m_latestGCI: 6964
050630  4:05:47 [Note] NDB Binlog: cluster failure for BANK3/ACCOUNT_TYPES.
050630  4:05:47 [Note] NDB Binlog: cluster failure for BANK3/SYSTEM_VALUES.
050630  4:05:47 [Note] NDB Binlog: cluster failure for BANK/ACCOUNT_TYPES.
050630  4:05:47 [Note] NDB Binlog: cluster failure for BANK/SYSTEM_VALUES.
050630  4:05:47 [Note] NDB Binlog: cluster failure for BANK/TRANSACTION.
050630  4:05:47 [Note] NDB Binlog: cluster failure for BANK/ACCOUNT.
050630  4:05:47 [Note] NDB Binlog: cluster failure for BANK/GL.
050630  4:05:47 [Note] NDB Binlog: cluster failure for test/t2.
050630  4:05:47 [Note] NDB Binlog: cluster failure for cluster_replication/apply_status.

http://bugs.mysql.com/bug.php?id=9961

What about the crash on node 3?