MySQL Bugs: #34893: falcon_bug_22207.test causes steady memory increase

Bug #34893	falcon_bug_22207.test causes steady memory increase
Submitted:	27 Feb 2008 16:37	Modified:	15 May 2009 12:56
Reporter:	Hakan Küçükyılmaz	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Falcon storage engine	Severity:	S3 (Non-critical)
Version:		OS:	Windows
Assigned to:	Kevin Lewis	CPU Architecture:	Any
Tags:	F_SCAVENGER

Description:
falcon_bug_22207.test fails on Pushbuild Windows.

How to repeat:
See Pushbuild for mysql-6.0-falcon and mysql-6.0-falcon-team tree on Windows.

falcon.falcon_bug_22207        [ fail ]

mysqltest: At line 52: query 'UPDATE t1 SET c2 = c2 + 1, c1 = concat(rpad(conv(c2,10,10),10,'_'), repeat(c3, 40))' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:
< snip >
SELECT c1 from t1;
c1
20000_____AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
20000_____BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
20000_____CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
SELECT c1 from t1;
c1
25000_____AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
25000_____BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
25000_____CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
SELECT c1 from t1;
c1
30000_____AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
30000_____BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
30000_____CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
SELECT c1 from t1;
c1
35000_____AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
35000_____BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
35000_____CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

More results from queries before failure can be found in e:\var-falcon-102\log\falcon_bug_22207.log

Stopping All Servers
Restoring snapshot of databases
Resuming Tests

Possibly connected to 34778

After increasing the middle loop from 500 to 10000 I can see a steady memory increase of mysqld. The test case states:
   # During this series of updates, there should be no increase in memory
   # since each previous record version should be freed once it is replaced.

Unfortunately there is an increase of memory usage.

This bug is no longer about falcon_bug_22207.test failing.  According to the  pushbuild xref tool it has not failed since 2008-04-09.  Around that time, Hakan changed the complain to a steady memory increase during the test.  He shortened the main test and added falcon_bug_22207-big.test.  

I just reran falcon_bug_22207-big and diagnosed the problem to the scavenger.  There are two types of record scavenging that should happen and they both occur in the same cycle or code section.

  1) Scavenge all Records and RecordVersions that are in the oldest generations once the total record cache size gets bigger than the scavenge threshold.  By default, this is 68% of 250Mb.   

  2) Scavenge all record versions that are no longer visible to any transaction.  This should be done every scavenge cycle.  But it only occurs along with the first type.

I observed that falcon_bug_22207-big will increase in memory until the falcon-record-scavenge-threshold is reached.  At that point, nearly all record cache memory was released.  This test only has 11 real records which are updated over and over.  The record cache just fills up with old RecordVersion objects.

Reassigning this to myself so since I have some other Scavenger changes I am working on.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/62571

2958 Kevin Lewis	2009-01-07
Bug#34893, Bug#36700, Bug#40342, Bug#40801
New scavenger code, See WL#4606, WL#4477 & WL#4478 for more info.
      
This is the second full patch of the new scavenger.  
1) Separate the two tasks involved in scavenging into
   A. Pruning old invisible record versions.
      1. As part of this task, a survey is taken of records
         in age groups so that the next step can be done in 
         one pass
   B. Retiring least-recently-used records from the record cache
2) Watch how much record space is added to the record cache 
   and start a new age group at regular intervals. 
3) Put the scavenger into a separate thread that can be 
   signaled when needed.

This patch contains many improvements over the last patch
including;
1) forcedRecordScavenge is now signalScavenge and the loop is 
   increased to 5 times attempting to get memory from the 
   record cache.
2) Unused code is deleted
3) Integrated with new dependency manager code so that it now
   uses Transaction::commitId to determine if a record is old
   enough to be scavenged.

Pushed into 6.0.10-alpha (revid:alik@sun.com-20090211182317-uagkyj01fk30p1f8) (version source revid:olav@sun.com-20090113103017-41jbad7qlvlwpwxh) (merge vers: 6.0.10-alpha) (pib:6)

An entry has been added to the 6.0.10 changelog: 

The Falcon storage engine has been updated to incorporate new code for the built-in scavenger service, which handles the caching of records in memory. This fixes a number of different issues related to the visibility of different records during certain operations and improves the memory usage.