Bug #28675 System error restarting node with full DataMemory
Submitted: 25 May 2007 12:46 Modified: 25 May 2012 11:59
Reporter: Geert Vanderkelen Email Updates:
Status: Not a Bug Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:5.0.40 OS:Linux
Assigned to: CPU Architecture:Any

[25 May 2007 12:46] Geert Vanderkelen
Description:
(logs will be attached)

2007-05-25
11:34		- Ran out of DataMemory
11:51:02	- Should have set DataMemory higher, but instead did IndexMemory (don't ask me why..)
11:51:02	- Started rolling restart: node 3 shut down
12:06:16	- Node 3 forced shutdown during start phase 5

By mistake I used 5.0.41 version instead of 5.0.40, but got same result.

Retried afterwards again with reverting IndexMemory and starting up the same version as node 4, but same error kept coming.

Kinda ugly error in ndb_mgm:

ndb_mgm> Node 3: Forced node shutdown completed. Occured during startphase 5. Initiated by signal 6. Caused by error 2303: 'System error, node killed during node restart by other node(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'. - Unknown error code: Unknown result: Unknown error code

Node 3 error log:

Time: Friday 25 May 2007 - 12:06:15
Status: Temporary error, restart node
Message: System error, node killed during node restart by other node (Internal error, programming error or missing error message, please report a bug)
Error: 2303
Error data: Killed by node 4 as copyfrag failed, error: 827
Error object: NDBCNTR (Line: 196) 0x0000000a
Program: ./libexec/ndbd
Pid: 25193
Trace: /data1/csc/16870/cluster/ndb_3_trace.log.1
Version: Version 5.0.40
***EOM***

Doing a system restart, the same node 3 goes down with the error.

How to repeat:
- I have no good repeatable test case.
- Some BLOB fields were used in the tables.

Following config was used, with 2 data nodes:

[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=2G
IndexMemory=300M

MaxNoOfOrderedIndexes = 256
MaxNoOfUniqueHashIndexes = 256

MaxNoOfConcurrentOperations = 300000
MaxNoOfConcurrentIndexOperations = 32000
NoOfFragmentLogFiles = 32
TimeBetweenLocalCheckpoints = 18
[25 Jun 2007 17:57] Hartmut Holzgraefe
Ndb error  827 is 

  NDB error code 827: Out of memory in Ndb Kernel, table data (increase DataMemory): Permanent error: Insufficient space

from your description it is not clear whether you incresed data memory 
when you reverted the index memory change ...
[1 Mar 2008 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[12 Apr 2008 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[16 May 2008 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[25 May 2012 11:59] Gustaf Thorslund
Old (now unsupported) version where it appears to have been misconfiguration. Closing as !bug, but might be a bit related to bug 30655:

  http://bugs.mysql.com/bug.php?id=30655