Bug #44375 Falcon CycleManager has a hot SyncObject

• Submitted: 21 Apr 2009 6:06
• Modified: 26 May 2010 17:46
• Reporter: Kevin Lewis
• Status: Unsupported
• Category: MySQL Server: Falcon storage engine
• Severity: S3 (Non-critical)
• Version: 6.0.10
• OS: Any
• CPU Architecture: Any

[21 Apr 2009 6:06] Kevin Lewis

Description:
Kelly Long reported that there are a slowdown at high concurrences (192 clients) using the new CycleManger.  It was suspected that the single SyncObject used by the CycleManager to pump threads out of the cycle is a hot spot because almost every thread in Falcon needs to lock this, which causes an interlocked increment, and associated cache line invalidation for other CPUs.  

To test this theory, I provided Kelly with code that uses an array of randomly assigned SyncObjects for each cycle.  Kelly reported that some of the recent slowdown he saw was gained back.  So this bug is created to track this performance drop using a single SyncObject per cycle.

How to repeat:
Run DBT2 before the Cycle manager, with it, and after this patch.

Suggested fix:
Create and use an array of randomly assigned SyncObjects for each cycle.

[21 Apr 2009 6:10] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/72542

3131 Kevin Lewis	2009-04-20
      Bug#44375 - Adding an array of SyncObjects to each cycle instead of only  
      one so that the cache line collisions at high concurrency are reduced.
      Create these SyncObjects as needed using CAS in case multiple threads 
      make them at the same time.

[23 Apr 2009 7:22] Bugs System

Pushed into 6.0.11-alpha (revid:[e-mail]-20090423071213-afmyrzvolemph7mz) (version source revid:[e-mail]-20090421195958-j33v1cuo3yer9niu) (merge vers: 6.0.11-alpha) (pib:6)

[30 Apr 2009 15:25] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/73198

2701 Kevin Lewis	2009-04-30
      Bug#44375 copntinued;  This is the second part of the optimization of the CycleManager 
      for high concurrency on multiple CPUs.  This patch spreads out the purgatory lists into 
      arrays or list heads and makes sure that each list head is on a separate cache line.
      
      In addition, it hashes the thread ID to find a consistent slot to use instead of calling rand() 
      to distribute access to the arrays evenly.  Since the threadID is not as evenly distributed as
      rand(), the array is made larger, at 256 elements instead of 64.  But this allows the same 
      thread to enter Falcon many times and keep using the same purgatories lists and Cycle 
      syncObject.
      
      Also, in order to make the CycleManager more responsive to the need to free memory, 
      itwill now skip the mandatory 1 second sleep between cycles if there is work to be done 
      after the previous cycle.  Then after a no-sleep cycle, if there is no work to be done the 
      next time, it will sleep less time than a full second.   Probably too much algorithm, but it is 
      adaptive.
      
      Also, the destructor now clears any uncleared purgatories, in case that were possible.
      second