Bug #13744 GCP Stop during GCP_COMMIT
Submitted: 4 Oct 2005 13:47 Modified: 7 Aug 2006 14:25
Reporter: Scott Tully Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:4.1.14, 5.1.10 OS:Linux (Redhat 9, FC4)
Assigned to: Jonathan Miller CPU Architecture:Any

[4 Oct 2005 13:47] Scott Tully 
Description:
Running 4.1.14 Cluster with 2 replica, 8 data nodes, and 3 api.  Just about once a day a data node disconnects without warning. It always seems to be the Master data node.  This is all i can find for a reason

Date/Time: Tuesday 4 October 2005 - 03:51:35
Type of error: error
Message: System error
Fault ID: 2303
Problem data: Node 11 killed this node because GCP stop was detected
Object of reference: NDBCNTR (Line: 193) 0x0000000c
ProgramName: /usr/local/mysql/libexec/ndbd
ProcessID: 25378
TraceFile: /var/lib/mysql-cluster/ndb_11_trace.log.11
Version 4.1.14
***EOM***

How to repeat:

Don't know.
[12 Oct 2005 13:31] Scott Tully 
It has not been an issue everyday since posting. Last night i shutdown the Cluster to make a config change and the same error occured again a few hours later.

Date/Time: Wednesday 12 October 2005 - 04:51:45
Type of error: error
Message: System error
Fault ID: 2303
Problem data: Node 11 killed this node because GCP stop was detected
Object of reference: NDBCNTR (Line: 193) 0x00000008
ProgramName: /usr/local/mysql/libexec/ndbd
ProcessID: 31202
TraceFile: /var/lib/mysql-cluster/ndb_11_trace.log.12
Version 4.1.14
***EOM***

Node 11 is killing itself. 

Trace log:  http://www.publicip.com/ndb_11_trace.log.12
[20 Oct 2005 4:43] Tomas Ulin 
Scott,

we are trying to hunt this issue down.  And we need the printout that comes in the ndb_*_out.log files.

Can you please provide them?

Thanks
[20 Oct 2005 13:18] Scott Tully 
oops, i attached the file like three hours ago, but didn't make a comment to update status... anyway, i think setting the TimeBetweenWatchDogCheck=30000 had a positive effect - but this has still happened even with that setting, just not as often.  Or it could be a coincedence and not related... 

Scott
[21 Apr 2006 18:56] Jonathan Miller 
ndb error report

Attachment: ndb_error_report_20060419154427.tar.bz2 (application/x-bzip, text), 191.48 KiB.
[7 Aug 2006 14:28] Jonathan Miller 
Retested and no failure found.