Bug #61132 Infinite loop in buf_page_get_gen() when handling compressed pages
Submitted: 11 May 2011 11:21 Modified: 21 Aug 2013 14:51
Reporter: Alexey Kopytov Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB Plugin storage engine Severity:S3 (Non-critical)
Version:5.1, 5.5 OS:Any
Assigned to: CPU Architecture:Any

[11 May 2011 11:21] Alexey Kopytov
Description:
After loading a compressed page in buf_page_get_gen() we allocate a new
block for decompression. The problem is that the compressed page is
neither buffer-fixed nor I/O-fixed by the time we call
buf_LRU_get_free_block(), so it may end up being evicted and returned
back as a new block.

We then call buf_page_hash_get() to see if the page hash has been
modified, and indeed it returns NULL because the original page in not in
the buffer pool anymore. In which case we go back to read the page
again. Here's the relevant code from buf_page_get_gen() in 5.1 plugin:

		/* Allocate an uncompressed page. */
		buf_pool_mutex_exit();
		mutex_exit(&buf_pool_zip_mutex);

		block = buf_LRU_get_free_block();
		ut_a(block);

		buf_pool_mutex_enter();
		mutex_enter(&block->mutex);

		{
			buf_page_t*	hash_bpage
				= buf_page_hash_get(space, offset);

			if (UNIV_UNLIKELY(bpage != hash_bpage)) {
				/* The buf_pool->page_hash was modified
				while buf_pool_mutex was released.
				Free the block that was allocated. */

				buf_LRU_block_free_non_file_page(block);
				mutex_exit(&block->mutex);

				block = (buf_block_t*) hash_bpage;
				goto loop2;
			}
		}

We had a reproducible case when the above loop becomes infinite with low
innodb_buffer_pool_size.

Temporarily buffer-fixing the compressed page to protect it from
buf_LRU_get_free_block() solved the problem.

How to repeat:
Examine the code in buf_page_get_gen().
[16 May 2011 19:18] Marko Mäkelä
While I have not repeated this bug, it is a plausible scenario.

We can prevent the eviction of the block by buffer-fixing it before releasing the buffer pool mutex. I/O-fixing the block earlier should not buy us anything. For example, buf_page_get_zip() can still interfere with buf_page_get_gen(). This should be extremely unlikely, because buf_page_get_zip() should only be used for accessing compressed BLOB pages, and compressed BLOB pages should never be accessed by buf_page_get_gen(), except when inserting the BLOB.
[16 May 2011 19:28] Marko Mäkelä
Tentative patch for mitigating the issue

Attachment: bug61132.patch (text/x-diff), 2.32 KiB.

[29 May 2013 8:22] Marko Mäkelä
Inaam Rana brought up an interesting point.

When running on a very small buffer pool, my fix could make the situation worse by buffer-fixing the compressed-only page while allocating an uncompressed page. If multiple threads are doing the same (for different blocks), we could run out of buffer pool.

However, I am not sure if this is a practical problem. The original hang should be possible with a reasonably-sized buffer pool as well, if the table sits idle long enough, so that the uncompressed pages will be evicted but the compressed pages will be retained in the buffer pool.

Without the fix, the very same page that buf_page_get_gen() is trying to decompress so that it can be accessed, could be evicted from the buffer pool, and buf_page_get_gen() would end up re-reading the same page from the data file.
[21 Aug 2013 14:51] Bugs System
Noted in 5.1.72, 5.5.34, 5.6.14, 5.7.3 changelog: 

An infinite loop could occur in "buf_page_get_gen" when handling
compressed-only pages. 

Thank you for the bug report.
[27 Sep 2013 7:28] Laurynas Biveinis
5.1$ bzr log -r 4038 -n0
------------------------------------------------------------
revno: 4038 [merge]
committer: Marko Mäkelä <marko.makela@oracle.com>
branch nick: mysql-5.1-current
timestamp: Wed 2013-08-21 11:54:09 +0300
message:
  Merge working copy to mysql-5.1.
    ------------------------------------------------------------
    revno: 4036.1.2 [merge]
    committer: Marko Mäkelä <marko.makela@oracle.com>
    branch nick: mysql-5.1
    timestamp: Wed 2013-08-21 10:03:31 +0300
    message:
      Merge mysql-5.1 to working copy.
    ------------------------------------------------------------
    revno: 4036.1.1
    committer: Marko Mäkelä <marko.makela@oracle.com>
    branch nick: mysql-5.1
    timestamp: Wed 2013-08-21 08:22:05 +0300
    message:
      Bug#12560151 61132: infinite loop in buf_page_get_gen() when handling
      compressed pages
      
      After loading a compressed-only page in buf_page_get_gen() we allocate a new
      block for decompression. The problem is that the compressed page is neither
      buffer-fixed nor I/O-fixed by the time we call buf_LRU_get_free_block(),
      so it may end up being evicted and returned back as a new block.
      
      buf_page_get_gen(): Temporarily buffer-fix the compressed-only block
      while allocating memory for an uncompressed page frame.
      This should prevent this form of the infinite loop, which is more likely
      with a small innodb_buffer_pool_size.
      
      rb#2511 approved by Jimmy Yang, Sunny Bains