Bug #68462 Server stuck in Waiting for Slave Workers to free pending events

• Submitted: 22 Feb 2013 10:59
• Modified: 28 May 2013 5:36
• Reporter: Martijn Otto
• Status: Closed
• Category: MySQL Server: Replication
• Severity: S1 (Critical)
• Version: 5.6.10
• OS: Linux
• CPU Architecture: Any

[22 Feb 2013 10:59] Martijn Otto

Description:
In some cases, replication gets stuck in the following state:

Waiting for Slave Workers to free pending events

We have set up a maximum of 16 worker threads. It only uses one however, as the master is still 5.5.29. All the worker threads are idling, while the master thread is stuck.

I am not sure which of these things actually triggers this bug.

How to repeat:
Enable row-based replication, with multiple worker threads and replicate a 5.5 master.

Suggested fix:
Don't get stuck in this state. All workers are idle, so there is nothing to wait for.

[22 Feb 2013 11:13] Martijn Otto

Note that the server is still running, so if necessary additional debugging can be performed.

[28 Feb 2013 11:26] Martijn Otto

Taking the server down after no response for 6 days.

[5 Mar 2013 17:49] Luis Soares

Thanks for the bug report. We are looking into it.

[5 Mar 2013 18:04] Luis Soares

Martijn,

  can you try to increase the size of slave-pending-jobs-size-max
(possibly to a big value, just for testing)? Do you still see the
same issue?

Thanks,
Luís

[28 May 2013 5:36] Jon Stephens

Thank you for your bug report. This issue has been committed to our source repository of that product and will be incorporated into the next release.

If necessary, you can access the source repository and build the latest available version, including the bug fix. More information about accessing the source trees is available at

    http://dev.mysql.com/doc/en/installing-source.html

[28 May 2013 5:41] Jon Stephens

Fixed in 5.6+. Documented as follows in the 5.6.12 and 5.7.2 changelogs:

        When the size of an execution event exceeded the maximum set for
        the buffer (slave_pending_jobs_size_max), row-based replication
        could hang with -Waiting for slave workers to free pending
        events-.

Closed.

[26 Jun 2014 19:07] Avijit Bhowmik

i am using mysql 5.6.17 community version for Red Hat Enterprise Linux Server release 5.8 and probably hitting this bug even with slave_pending_jobs_size_max (23068672) in slave set higher than max_allowed_packet (20971520) in master.
2014-06-26 18:53:28 29048 [Note] Multi-threaded slave: Coordinator has waited 701 times hitting slave_pending_jobs_size_max; current event size = 8167.
2014-06-26 18:56:43 29048 [Note] Multi-threaded slave: Coordinator has waited 711 times hitting slave_pending_jobs_size_max; current event size = 7813.
2014-06-26 19:00:01 29048 [Note] Multi-threaded slave: Coordinator has waited 721 times hitting slave_pending_jobs_size_max; current event size = 8193.
2014-06-26 19:03:41 29048 [Note] Multi-threaded slave: Coordinator has waited 731 times hitting slave_pending_jobs_size_max; current event size = 8176.