Bug #6616 MySQL server 100% CPU if FLUSH TABLES WITH READ LOCK + INSERT
Submitted: 14 Nov 2004 21:26 Modified: 7 May 2005 11:41
Reporter: Guilhem Bichot Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:4.0 OS:Linux (Linux)
Assigned to: Antony Curtis CPU Architecture:Any

[14 Nov 2004 21:26] Guilhem Bichot
Description:
Observed by adding a sleep(10) in FLUSH TABLES WITH READ LOCK between the lock_global_read_lock() and the close_cached_tables() in sql_parse.cc:
      if (lock_global_read_lock(thd))
	return 1;
      sql_print_error("please do the INSERT now, sleeping 10 secs\n");
      sleep(10);
      result=close_cached_tables(thd,(options & REFRESH_FAST) ? 0 : 1,
                                 tables);

Then did FLUSH TABLES WITH READ LOCK in connection1, while it slept, did INSERT INTO TABLE t VALUES(1) in connection2 (using mysql -A). After 10 secs the FLUSH returned, the INSERT stayed blocked (normal, by the FLUSH) but CPU went to 98% forever.

How to repeat:
see description, you need a sleep().

Suggested fix:
I can only show the infinite loop: mysql_insert() calls mysql_lock_tables() which calls
wait_if_global_read_lock(); in this function the place where the thread should block is:

    while (must_wait && ! thd->killed &&
	   (!abort_on_refresh || thd->version == refresh_version))
      (void) pthread_cond_wait(&COND_refresh,&LOCK_open);
gdb:
must_wait=1 normal
thd->killed=0 normal
abort_on_refresh=1 that's how code is
thd->version == refresh_version : false (...<... is true)
so it does not enter the cond_wait(), returns 0 immediately to caller (so function does not what it's supposed to do: it's supposed to WAIT until global read lock is gone), and later caller loops ("goto retry") etc.
This bug must have been existing for a quite long time (the only recent change is the introduction by me of must_wait but it cannot be the cause here).
[20 Nov 2004 20:40] Antony Curtis

 
[16 Dec 2004 8:52] Ingo Strüwing
It seems like I'm not allowed to do reviews. But I try anyway.
[21 Dec 2004 10:08] Ingo Strüwing
The patch cures the infinite loop. Nevertheless I ask for more investigation, as I noticed a hang during my tests.
[10 Mar 2005 23:30] Antony Curtis
Ingo has reservations with respect to the patch and would like to run tests to validate (or break) its effectiveness.
[21 Apr 2005 5:50] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/24182
[21 Apr 2005 5:52] Antony Curtis
Ingo was unable to reproduce a hang with patch applied.
[27 Apr 2005 9:00] Antony Curtis
Patch has been examined by Ingo and Guilhem
[6 May 2005 10:12] Michael Widenius
Patch will not solve problem. Discussed with Antony and proposed a better solution which he will test
[7 May 2005 11:41] Antony Curtis
Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html