Bug #21137 Limit max. size of allocated page chunks
Submitted: 19 Jul 2006 8:48 Modified: 25 May 2007 15:27
Reporter: Hartmut Holzgraefe Email Updates:
Status: Won't fix Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:4.1, 5.0, 5.1 OS:Any
Assigned to: Hartmut Holzgraefe CPU Architecture:Any

[19 Jul 2006 8:48] Hartmut Holzgraefe 
Description:
DataMemory pages for cluster table fragments are allocated in chunks of about 19% of the current table size, trying to keep the total number of chunks to be searched low. This sort of backfires with really large tables as e.g. the next extend of a table that already takes about 80% of the DataMemory will consume *all* remaining free pages no matter what...

How to repeat:
Create a table and fill it with data, watch the ALL DUMP 1000 output of free pages while the table fills up, see that the number of allocated pages grows in chunks that grow by about 19% every time a new chunk needs to be allocated

Suggested fix:
Put an upper limit on the number of pages allocated in one chunk, 
e.g. 1024 pages (32MB DataMemory) or maybe even something configurable. 
This will increase the number of chunks for large tables a bit (by about 30% for a table with 1GB fragments) but IMHO that is still a good tradeoff
[1 Aug 2006 15:28] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/9905

ChangeSet@1.2248, 2006-08-01 17:27:44+02:00, hartmut@mysql.com +1 -0
  limit the max. size of allocated page chunks (Bug #21137)
  
  motivation: with the current 18% increase a table that already consumes 
  about 80% of DataMemory is likely to allocate *all* remaining RAM on 
  the next extend. Limiting the max. extend chunk size to 1024 pages 
  will increase the number of chunks for a 1GB table fragment by about 20%
  but will lead to a way more nice behavior when tables grow big and memory
  starts to become tight