Bug #41853 Forced node shutdown completed. Caused by error 2303: 'System error, node killed
Submitted: 5 Jan 2009 3:41 Modified: 8 Jan 2009 14:05
Reporter: Alex Loo Wai Mun Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S2 (Serious)
Version:mysql-5.1.23 ndb-6.2.15 OS:Linux (Enterprise 5.1)
Assigned to: CPU Architecture:Any

[5 Jan 2009 3:41] Alex Loo Wai Mun
Description:
Currently I facing one of the node storage forced shutdown suddently under production environment.

ndb_2_out.log
2009-01-01 15:58:48 [ndbd] INFO     -- Node 2 killed this node because GCP stop was detected
2009-01-01 15:58:48 [ndbd] INFO     -- NDBCNTR (Line: 235) 0x0000000a
2009-01-01 15:58:48 [ndbd] INFO     -- Error handler shutting down system
2009-01-01 15:58:48 [ndbd] INFO     -- Error handler shutdown completed - exiting
2009-01-01 15:58:48 [ndbd] ALERT    -- Node 2: Forced node shutdown completed. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

I believed that this bug has been reported by others before. Not sure my problem same as them or not. The most similar case was Bug #37227

Disk table using.

How to repeat:
Unknown

Suggested fix:
Start back the ndbd at node 2
[5 Jan 2009 3:44] Alex Loo Wai Mun
node 2 out log

Attachment: ndb_2_out.zip (application/zip, text), 23.24 KiB.

[5 Jan 2009 3:44] Alex Loo Wai Mun
node 2 trace log

Attachment: ndb_2_trace.log.zip (application/zip, text), 46.93 KiB.

[5 Jan 2009 3:57] Alex Loo Wai Mun
mysql-cluster management configuration file

Attachment: config.ini (application/octet-stream, text), 612 bytes.

[8 Jan 2009 14:05] Martin Skold
Duplicate of Bug#37227