MySQL Bugs: #28675: System error restarting node with full DataMemory

Bug #28675	System error restarting node with full DataMemory
Submitted:	25 May 2007 12:46	Modified:	25 May 2012 11:59
Reporter:	Geert Vanderkelen	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S3 (Non-critical)
Version:	5.0.40	OS:	Linux
Assigned to:		CPU Architecture:	Any

Description:
(logs will be attached)

2007-05-25
11:34 - Ran out of DataMemory
11:51:02 - Should have set DataMemory higher, but instead did IndexMemory (don't ask me why..)
11:51:02 - Started rolling restart: node 3 shut down
12:06:16 - Node 3 forced shutdown during start phase 5

By mistake I used 5.0.41 version instead of 5.0.40, but got same result.

Retried afterwards again with reverting IndexMemory and starting up the same version as node 4, but same error kept coming.

Kinda ugly error in ndb_mgm:

ndb_mgm> Node 3: Forced node shutdown completed. Occured during startphase 5. Initiated by signal 6. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. - Unknown error code: Unknown result: Unknown error code

Node 3 error log:

Time: Friday 25 May 2007 - 12:06:15
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Killed by node 4 as copyfrag failed, error: 827
Error object: NDBCNTR (Line: 196) 0x0000000a
Program: ./libexec/ndbd
Pid: 25193
Trace: /data1/csc/16870/cluster/ndb_3_trace.log.1
Version: Version 5.0.40
***EOM***

Doing a system restart, the same node 3 goes down with the error.

How to repeat:
- I have no good repeatable test case.
- Some BLOB fields were used in the tables.

Following config was used, with 2 data nodes:

[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=2G
IndexMemory=300M

MaxNoOfOrderedIndexes = 256
MaxNoOfUniqueHashIndexes = 256

MaxNoOfConcurrentOperations = 300000
MaxNoOfConcurrentIndexOperations = 32000
NoOfFragmentLogFiles = 32
TimeBetweenLocalCheckpoints = 18

Ndb error  827 is 

  NDB error code 827: Out of memory in Ndb Kernel, table data (increase DataMemory): Permanent error: Insufficient space

from your description it is not clear whether you incresed data memory 
when you reverted the index memory change ...

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Old (now unsupported) version where it appears to have been misconfiguration. Closing as !bug, but might be a bit related to bug 30655:

  http://bugs.mysql.com/bug.php?id=30655