Bug #60939 Infinite loop in shutdown with dedicated purge threads
Submitted: 21 Apr 2011 3:31 Modified: 27 Apr 2011 2:40
Reporter: Laurynas Biveinis (OCA) Email Updates:
Status: Duplicate Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:5.6.2 OS:Linux
Assigned to: CPU Architecture:Any

[21 Apr 2011 3:31] Laurynas Biveinis
Description:
Server with innodb_fast_shutdown=0 and innodb_purge_threads=4 enters infinite loop in purge coordinator thread if there is a significant purge lag on shutdown.

The error log shows many copies of the following message:
110420 20:06:11  InnoDB: Waiting for master thread  to be suspended

Thread 5 (full GDB backtraces in attachment) is looping around waiting for purge workers to complete, but there are no purge workers running.

    (gdb) print srv_get_task_queue_length()
    $1 = 3
    (gdb) print purge_sys->n_executing
    $3 = 0
    (gdb) print purge_sys->n_submitted
    $4 = 264148
    (gdb) print purge_sys->n_completed
    $5 = 264145

    Thread 5 (Thread 0x4a12b940 (LWP 32483)):
    #0  0x0000003fcc4cd212 in select () from /lib64/libc.so.6
    #1  0x00000000009210af in os_thread_sleep (tm=<value optimized out>) at /data/laurynas/upstream/mysql-5.6.2-m5/storage/innobase/os/os0thread.c:274
    #2  0x00000000008648c5 in trx_purge_wait_for_workers_to_complete (n_purge_threads=4, batch_size=20)
    at /data/laurynas/upstream/mysql-5.6.2-m5/storage/innobase/trx/trx0purge.c:1164
    #3  trx_purge (n_purge_threads=4, batch_size=20) at /data/laurynas/upstream/mysql-5.6.2-m5/storage/innobase/trx/trx0purge.c:1288
    #4  0x000000000085968d in srv_purge_coordinator_thread (arg=<value optimized out>) at /data/laurynas/upstream/mysql-5.6.2-m5/storage/innobase/srv/srv0srv.c:2831
    #5  0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
    #6  0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

How to repeat:
In description.
[21 Apr 2011 3:32] Laurynas Biveinis
GDB backtraces

Attachment: 60939-backtraces.txt (text/plain), 13.88 KiB.

[21 Apr 2011 3:32] Laurynas Biveinis
This could be duplicate of http://bugs.mysql.com/bug.php?id=58868 , but I have no way of telling.
[22 Apr 2011 14:40] Valeriy Kravchuk
Let's assume this is a duplicate of bug #58868 for now.
[27 Apr 2011 2:40] Laurynas Biveinis
This issue becomes unobservable with the patch applied that removes purge coordinator sleeps.
[27 Apr 2011 2:42] Laurynas Biveinis
Patch to remove purge sleeps

Attachment: 5.6-purge-fix.patch (text/x-diff), 748 bytes.

[27 Apr 2011 5:52] Sunny Bains
Hi,

In the latest version of the unreleased code there are no sleeps. The code
is now similar to the way 5.5 purge thread works, using os_event_t.

Regards,
-sunny