Bug #75128 Excessive memory usage: NdbScanOperation::close() does not release buffer mem.

• Submitted: 6 Dec 2014 12:22
• Modified: 12 Jan 2015 18:35
• Reporter: Ole John Aske
• Status: Closed
• Category: MySQL Cluster: NDB API
• Severity: S3 (Non-critical)
• Version: 7.1.34
• OS: Any
• CPU Architecture: Any

[6 Dec 2014 12:22] Ole John Aske

Description:
The NdbScanOperation allocate a possible huge buffer for receiving
the scanned rows. This buffer is allocated with size:

  'Max row size including all columns, even if not selected, with
   varchar expanded to max size.'
*
   Max rows in batch
*
   Max 'parallelism' for the query

This can easily be multiple megabytes of memory, and we don't want to
keep this allocated longer than necessary. Typically a ::close()
on the NdbScan cursor is a place where we would expect that the
such buffers are release.

However, it turns out that this buffer is kept until the NdbScanOperation
itself is 'release'. This doesn't happen until the NdbTransaction 
'owning' the ScanOperation is closed.

This could lead to excessive memory usage in an application where
multiple scans are created within the same transaction - even if
the scans are closed at the end of their lifecycle

How to repeat:

testIndex -n SR1_O

Suggested fix:
Let NdbScanOperation::close() release the scan buffer memory

[8 Dec 2014 10:09] Ole John Aske

Posted by developer:
 
When documenting, make sure this is covered:

- There is not a memory leak as such, but the memory
  allocated to hold the scan result set is kept longer
  than required.

- This memory used to be held until the NdbScanOperation object
  was releases - Either by closing the transaction, or
  calling NdbScanOperation::close(, release==true).
  It is now released when the 'cursor' navigating the
  result set is closed by calling NdbScanOperation::close().
  (Independent of the value of the 'release' argument)

- If not already clear from the documentation, it should
  be made clear that any rows returned from a scan is not
  valid after a NdbScanOperation::close()

[12 Jan 2015 18:35] Jon Stephens

Documented fix in the NDB 7.1.34, 7.2.19, and 7.3.8 changelogs as follows:

    The buffer allocated by an NdbScanOperation for receiving
    scanned rows was not released until the NdbTransaction owning
    the scan operation was closed. This could lead to excessive
    memory usage in an application where multiple scans were created
    within the same transaction, even if these scans were closed at
    the end of their lifecycle, unless NdbScanOperation::close() was
    invoked with the releaseOp argument equal to true. Now the
    buffer is released whenever the cursor navigating the result set
    is closed with NdbScanOperation::close(), regardless of the
    value of this argument.
      
Closed.