Bug #75599 | ScanOperation allocate to much memory for their receive buffers | ||
---|---|---|---|
Submitted: | 23 Jan 2015 11:49 | Modified: | 13 Mar 2015 13:53 |
Reporter: | Ole John Aske | Email Updates: | |
Status: | Closed | Impact on me: | |
Category: | MySQL Cluster: Cluster (NDB) storage engine | Severity: | S3 (Non-critical) |
Version: | 7.1.34 | OS: | Any |
Assigned to: | CPU Architecture: | Any |
[23 Jan 2015 11:49]
Ole John Aske
[13 Mar 2015 13:53]
Jon Stephens
Documented fix as follows in the NDB 7.1.35, 7.2.20, 7.3.9, and 7.4.5 changelogs: A scan operation, whether it is a single table scan or a query scan used by a pushed join, stores the result set in a buffer. This maximum size of this buffer is calculated and preallocated before the scan operation is started. This buffer may consume a considerable amount of memory; in some cases we have observed a 2 GB buffer footprint in tests that executed 100 parallel scans with 2 single-threaded (ndbd) data nodes. Memory consumption was found to scale linearly with additional fragments. A number of root causes were discovered that led to this problem: -Result rows were unpacked to full NdbRecord format before they were stored in the buffer. If only some but not all columns of a table were selected, the buffer contained empty space (essentially wasted). -Due to the buffer format being unpacked, VARCHAR and VARBINARY columns had to be allocated for the maximum size defined for such columns. -BatchByteSize and MaxScanBatchSize values were not taken into consideration as a limiting factor when calculating the maximum buffer size. These issues became more evident in NDB 7.2 and later MySQL Cluster release series. This was due to the fact buffer size is scaled by BatchSize, and that the default value for this parameter was increased fourfold (from 64 to 256) beginning with MySQL Cluster NDB 7.2.1. This fix causes result rows to be buffered using the packed format instead of the unpacked format; a buffered scan result row is now not unpacked until it becomes the current row. In addition, BatchByteSize and MaxScanBatchSize are now used as limiting factors when calculating the required buffer size. Also as part of this fix, refactoring has been done to separate handling of buffered (packed) from handling of unbuffered result sets, and to remove code that had been unused since NDB 7.0 or earlier. The NdbRecord class declaration has also been cleaned up by removing a number of unused or redundant member variables. Closed.