Bug #64491 mysql server using cpu 100% when all threads are idle
Submitted: 29 Feb 2012 9:48 Modified: 17 Apr 2012 10:38
Reporter: jiang xiaobing Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server Severity:S2 (Serious)
Version:5.5.20, 5.5.22 OS:Linux (centos 2.6.18-274.12.1.e15)
Assigned to: CPU Architecture:Any

[29 Feb 2012 9:48] jiang xiaobing 
Description:
one thread call sched_yield(), but other threads are't ready run. so cause cpu 100%.

[Switching to thread 3 (Thread 0x497bd940 (LWP 31766))]#0  0x000000361eabb187 in sched_yield () from /lib64/libc.so.6
(gdb) bt
#0  0x000000361eabb187 in sched_yield () from /lib64/libc.so.6
#1  0x00000000008cb75d in _lf_pinbox_put_pins (pins=0x25133c0) at /home/mysql/soft/mysql-5.5.20/mysys/lf_alloc-pin.c:233
#2  0x00000000007bc792 in lf_pinbox_put_pins (pfs=0x2aaac0f840c0) at /home/mysql/soft/mysql-5.5.20/include/lf.h:172
#3  destroy_thread (pfs=0x2aaac0f840c0) at /home/mysql/soft/mysql-5.5.20/storage/perfschema/pfs_instr.cc:793
#4  0x00000000004f672c in cache_thread (thd=<value optimized out>, put_in_cache=true)
    at /home/mysql/soft/mysql-5.5.20/sql/mysqld.cc:2126
#5  one_thread_per_connection_end (thd=<value optimized out>, put_in_cache=true)
    at /home/mysql/soft/mysql-5.5.20/sql/mysqld.cc:2195
#6  0x00000000005ec3bd in do_handle_one_connection (thd_arg=0x2ab9402a3120)
    at /home/mysql/soft/mysql-5.5.20/sql/sql_connect.cc:796
#7  0x00000000005ec4ea in handle_one_connection (arg=<value optimized out>)
    at /home/mysql/soft/mysql-5.5.20/sql/sql_connect.cc:708
#8  0x00000000007c4937 in pfs_spawn_thread (arg=<value optimized out>)
    at /home/mysql/soft/mysql-5.5.20/storage/perfschema/pfs.cc:1015
#9  0x000000361f60673d in start_thread () from /lib64/libpthread.so.0
#10 0x000000361ead44bd in clone () from /lib64/libc.so.6

(gdb) info threads

  32 Thread 0x40a90940 (LWP 16028)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  31 Thread 0x428ee940 (LWP 16029)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  30 Thread 0x432ef940 (LWP 16030)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  29 Thread 0x43cf0940 (LWP 16031)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  28 Thread 0x446f1940 (LWP 16032)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  27 Thread 0x450f2940 (LWP 16033)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  26 Thread 0x45af3940 (LWP 16034)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  25 Thread 0x464f4940 (LWP 16035)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  24 Thread 0x46ef5940 (LWP 16036)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  23 Thread 0x478f6940 (LWP 16037)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  22 Thread 0x482f7940 (LWP 16043)  0x000000361f60b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  21 Thread 0x48cf8940 (LWP 16044)  0x000000361f60b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  20 Thread 0x496f9940 (LWP 16045)  0x000000361f60b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19 Thread 0x41d86940 (LWP 16046)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  18 Thread 0x410c9940 (LWP 16047)  0x000000361f60e898 in do_sigwait () from /lib64/libpthread.so.0
  17 Thread 0x49881940 (LWP 6747)  0x000000361f60d97b in read () from /lib64/libpthread.so.0
  16 Thread 0x41047940 (LWP 12362)  0x000000361f60d97b in read () from /lib64/libpthread.so.0
  15 Thread 0x41de8940 (LWP 9079)  0x000000361f60d97b in read () from /lib64/libpthread.so.0
  14 Thread 0x498b2940 (LWP 14266)  0x000000361f60d97b in read () from /lib64/libpthread.so.0
  13 Thread 0x4972a940 (LWP 14318)  0x000000361f60d97b in read () from /lib64/libpthread.so.0
  12 Thread 0x40b23940 (LWP 6887)  0x000000361f60d97b in read () from /lib64/libpthread.so.0
  11 Thread 0x41154940 (LWP 31236)  0x000000361f60d524 in __lll_lock_wait () from /lib64/libpthread.so.0
  10 Thread 0x41db7940 (LWP 31434)  0x000000361f60d524 in __lll_lock_wait () from /lib64/libpthread.so.0
  9 Thread 0x41e19940 (LWP 31436)  0x000000361f60d524 in __lll_lock_wait () from /lib64/libpthread.so.0
  8 Thread 0x4981f940 (LWP 31566)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7 Thread 0x411fe940 (LWP 31568)  0x000000361f60d524 in __lll_lock_wait () from /lib64/libpthread.so.0
  6 Thread 0x40ac1940 (LWP 31636)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5 Thread 0x497ee940 (LWP 31638)  0x000000361f60d524 in __lll_lock_wait () from /lib64/libpthread.so.0
  4 Thread 0x4122f940 (LWP 31703)  0x000000361f60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3 Thread 0x497bd940 (LWP 31766)  0x000000361eabb187 in sched_yield () from /lib64/libc.so.6
  2 Thread 0x40c37940 (LWP 31768)  0x000000361f60d524 in __lll_lock_wait () from /lib64/libpthread.so.0
* 1 Thread 0x2ba952999610 (LWP 16026)  0x000000361f60d524 in __lll_lock_wait () from /lib64/libpthread.so.0

How to repeat:
can't repeat manually.
[4 Mar 2012 17:32] Valeriy Kravchuk 
It would be nice to see the output of SHOW FULL PROCESSLIST next time this happens, or get a core file of mysqld in this state.
[5 Mar 2012 2:43] jiang xiaobing 
when it happens, the client can't connect to the server. and the coredump will be very big. maybe I can using gdb log bt info using "thread apply all bt"
[6 Mar 2012 7:11] Valeriy Kravchuk 
Yes, the output of "thread apply all bt" from gdb (compressed) would be useful.
[4 Apr 2012 13:38] Andreas Eiken 
same problem with 5.5.22

Attachment: mysqlreport (application/octet-stream, text), 169.18 KiB.
[13 Apr 2012 17:12] jiang xiaobing 
D.1.2. Changes in MySQL 5.5.23

An infinite thread loop could develop within Performance Schema, causing the server to become unresponsive. (Bug #13898343)

may be the same bug, but where can find the Bug #13898343   ??????

after google, find that:
http://lists.mysql.com/commits/143350?f=plain
[14 Apr 2012 11:30] Valeriy Kravchuk 
This is a reference to a bug number in internal Oracle bugs database. Just check if upgrade to 5.5.23 solves the problem for you.
[17 Apr 2012 10:35] Andreas Eiken 
I think, this patch solved the problem.
I've done my tests again as followed, every test was done 10 times:
5.5.22 with performance schema: server stalled during the third test.
5.5.22 without performance schema: tests finished without server errors.
5.5.23 with performance schema: tests finished without server errors.
[17 Apr 2012 10:37] MySQL Verification Team 
Well, this is then a duplicate of the bug I filed:
Bug 13898343 - THREAD LOOPS ENDLESSLY IN LF_PINBOX_PUT_PINS WHILE HOLDING LOCK_THREAD_COUNT

Fixed in 5.5.23.