Bug #52502 Performance schema does not start with large mutex_instance buffers
Submitted: 31 Mar 2010 12:56 Modified: 4 Aug 2010 20:06
Reporter: Marc ALFF Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Performance Schema Severity:S1 (Critical)
Version:mysql-trunk (5.5.3) OS:Any
Assigned to: Marc ALFF CPU Architecture:Any

[31 Mar 2010 12:56] Marc ALFF 
Description:
Seen while instrumenting code that uses a large (250,000) mutex_instance number:

The server fails to start, spending time in create_mutex() and create_rwlock().

How to repeat:

Requires code changes, to instrument code using a large number of mutex instances.
[31 Mar 2010 13:29] Marc ALFF 
Analysis

When trying to create a large number of mutexes, say, 750,000,
in a buffer sized to hold 250,000, create_mutex() loops forever.

Assuming the first 250,000 mutex creation works, the mutex_array is now full.

For each of the remaining 500,000 calls, create_mutex() will scan the entire
buffer of size 250,000 ... 

The code needs to be more robust and fail sooner, by incrementing the lost counters, when a large buffer is used and is full.

Also, the randomize_index() function needs improvements : the buffer may not be full but only appear to be full, due to bad statistical spread of the not-so-random index.
[31 Mar 2010 15:49] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/104734

3144 Marc Alff	2010-03-31
      Bug#52502 Performance schema does not start with large mutex_instance buffers
      
      This is a performance issue affecting scalability.
      
      Prior to this fix, the performance schema code did not scale well
      when a very large number of instruments are created in the code.
      
      The root cause was create_mutex() spinning "forever",
      trying to find an empty slot in a full mutex_array.
      
      With this fix:
      - functions like create_mutex() abort sooner,
        when used with a very large array which is full or almost full.
      - randomize_index() has been revised to improve
        the statistical spread of values in the mutex_array.  
        See the comments in the code for details.
      
      Other allocation functions similar to create_mutex()
      have been also fixed.
      
      No MTR test case provided (performance scalability issue).
      Fix tested manually.
[1 Apr 2010 8:08] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/104765

3144 Marc Alff	2010-04-01
      Bug#52502 Performance schema does not start with large mutex_instance buffers
      
      This is a performance issue affecting scalability.
      
      Prior to this fix, the performance schema code did not scale well
      when a very large number of instruments are created in the code.
      
      The root cause was create_mutex() spinning "forever",
      trying to find an empty slot in a full mutex_array.
      
      With this fix:
      - functions like create_mutex() abort sooner,
        when used with a very large array which is full or almost full.
      - randomize_index() has been revised to improve
        the statistical spread of values in the mutex_array.  
        See the comments in the code for details.
      
      Other allocation functions similar to create_mutex()
      have been also fixed.
      
      No MTR test case provided (performance scalability issue).
      Fix tested manually.
[1 Apr 2010 13:31] Guilhem Bichot 
approved with suggestions sent by mail.
[1 Apr 2010 14:27] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/104912

3144 Marc Alff	2010-04-01
      Bug#52502 Performance schema does not start with large mutex_instance buffers
      
      This is a performance issue affecting scalability.
      
      Prior to this fix, the performance schema code did not scale well
      when a very large number of instruments are created in the code.
      
      The root cause was create_mutex() spinning "forever",
      trying to find an empty slot in a full mutex_array.
      
      With this fix:
      - functions like create_mutex() abort sooner,
        when used with a very large array which is full or almost full.
      - randomize_index() has been revised to improve
        the statistical spread of values in the mutex_array.  
        See the comments in the code for details.
      
      Other allocation functions similar to create_mutex()
      have been also fixed.
      
      No MTR test case provided (performance scalability issue).
      Fix tested manually.
[1 Apr 2010 14:29] Marc ALFF 
Re posted a clean patch with code review fixes.
[13 Apr 2010 10:07] Marc ALFF 
Pushed into:
- mysql-next-mr-bugfixing
- mysql-6.0-codebase-bugfixing
[27 Apr 2010 9:46] Bugs System 
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100427094135-5s49ecp3ckson6e2) (version source revid:alik@sun.com-20100427093843-uekr85qkd7orx12t) (merge vers: 6.0.14-alpha) (pib:16)
[27 Apr 2010 9:51] Bugs System 
Pushed into mysql-next-mr (revid:alik@sun.com-20100427094036-38frbg3famdlvjup) (version source revid:alik@sun.com-20100427093825-92wc8b22d4yg34ju) (pib:16)
[8 May 2010 22:03] Paul DuBois 
Noted in 6.0.14 changelog.

Performance Schema could enter an infinite loop if required to create
a large number of mutex instances.
[15 Jul 2010 23:53] Marc ALFF 
The bug is tagged SRGAQUAL,SR55RC, SRMR03, but the fix is missing from 5.5

Changing to need merge
[16 Jul 2010 0:03] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/113717

3117 Marc Alff	2010-07-15
      Bug#52502 Performance schema does not start with large mutex_instance buffers
      
      Backport from mysql-next-mr (5.6) to mysql-trunk (5.5)
[16 Jul 2010 0:33] Marc ALFF 
Pushed into mysql-trunk-bugfixing (5.5.6)
[22 Jul 2010 16:10] Paul DuBois 
Noted in 5.5.6 changelog.
[23 Jul 2010 12:26] Bugs System 
Pushed into mysql-trunk 5.5.6-m3 (revid:alik@sun.com-20100723121820-jryu2fuw3pc53q9w) (version source revid:vasil.dimov@oracle.com-20100531152341-x2d4hma644icamh1) (merge vers: 5.5.5-m3) (pib:18)
[23 Jul 2010 12:33] Bugs System 
Pushed into mysql-next-mr (revid:alik@sun.com-20100723121929-90e9zemk3jkr2ocy) (version source revid:vasil.dimov@oracle.com-20100531152341-x2d4hma644icamh1) (pib:18)
[4 Aug 2010 8:08] Bugs System 
Pushed into mysql-trunk 5.6.1-m4 (revid:alik@ibmvm-20100804080001-bny5271e65xo34ig) (version source revid:alik@sun.com-20100427095914-pzlxbqjjtnngxmf0) (merge vers: 5.6.99-m4) (pib:18)
[4 Aug 2010 8:23] Bugs System 
Pushed into mysql-trunk 5.6.1-m4 (revid:alik@ibmvm-20100804081533-c1d3rbipo9e8rt1s) (version source revid:alik@sun.com-20100427095914-pzlxbqjjtnngxmf0) (merge vers: 5.6.99-m4) (pib:18)
[4 Aug 2010 20:06] Paul DuBois 
Bug is not present in any released 5.6.x version.