Description:
The connection accept loop and the shutdown path have a classic check-then-sleep race
that can make mysqld hang forever during shutdown.
Accept loop (main thread), sql/conn_handler/connection_acceptor.h:64:
while (!connection_events_loop_aborted()) {
Channel_info *channel_info = m_listener->listen_for_connection_event();
...
}
Mysqld_socket_listener::listen_for_connection_event() then blocks in an
infinite-timeout poll, sql/conn_handler/socket_connection.cc:1351:
int retval = poll(&m_poll_info.m_fds[0], m_socket_vector.size(), -1);
Shutdown path (signal handling thread), sql/mysqld.cc:3902-3924:
if (!connection_events_loop_aborted()) {
set_connection_events_loop_aborted(true);
...
mysql_mutex_lock(&LOCK_socket_listener_active);
while (socket_listener_active) {
if (pthread_kill(main_thread_id, SIGALRM)) { assert(false); break; }
mysql_cond_wait(&COND_socket_listener_active, &LOCK_socket_listener_active);
}
mysql_mutex_unlock(&LOCK_socket_listener_active);
close_connections();
}
The wake-up signal is handled by an empty handler (sql/mysqld.cc:3699,
installed without SA_RESTART at sql/mysqld.cc:3746-3748), whose only purpose
is to interrupt poll() with EINTR.
The race window:
1. Main thread evaluates `while (!connection_events_loop_aborted())` -> false
(flag not set yet).
2. Before the main thread enters poll(), the shutdown thread sets the flag to
true and sends the wake-up signal (SIGUSR1 in 5.7 and 8.0.0-8.0.18;
SIGALRM since 8.0.19, per WL-13689).
3. The empty signal handler runs on the main thread and returns; the signal
is consumed *before* poll() starts, so nothing will ever interrupt the
upcoming poll(-1) call.
4. Main thread enters poll(-1). If no new connection ever arrives (typical
once shutdown begins and clients have drained), poll sleeps forever.
Resulting deadlock:
- Main thread: blocked forever in poll(-1).
- Signal thread: sends the signal exactly once, then blocks forever in
mysql_cond_wait() on COND_socket_listener_active, waiting for the main
thread to leave the accept loop (socket_listener_active=false is only set
after the loop exits, sql/mysqld.cc:10299-10303).
Shutdown never completes; mysqld must be killed with SIGKILL.
The atomic abort flag (renamed from `abort_loop` and made atomic by
BUG#2013467) only guarantees *visibility* of the flag; it cannot make the
flag-check + poll-enter sequence atomic, so it does not close this window.
The LOCK_socket_listener_active/COND handshake is only a rendezvous for the
normal case (signal arrives while poll is blocked -> EINTR -> loop re-checks
flag); it sends the signal only once and does not retry, so it does not
mitigate a lost signal either.
Any later connection attempt (even a failed TCP connect / health-check
probe) wakes poll(), the loop sees the flag and exits, so in busy/ probed
deployments the hang self-heals. On a quiet server the hang is permanent.
The separate admin-interface listener thread (handle_admin_socket(),
socket_connection.cc:1080-1082, woken by pthread_kill(admin_socket_thread_id,
SIGALRM) in close_listener(), socket_connection.cc:1468) has the identical
pattern and the identical race.
Note: this mechanism has been unchanged since WL#7260 (2013). WL-13689
(8.0.19) only repurposed SIGUSR1 -> SIGALRM for the wake-up ("SIGALRM will
replace SIGUSR1 for all its previous uses"); the race window is unaffected.
How to repeat:
The un-instrumented race window is only a few instructions wide, so a
deterministic repro widens it with a small debug patch that simulates a
scheduler preemption at exactly the wrong spot (the same thing a context
switch can do in production).
1. Apply this instrumentation patch to mysqld and rebuild:
--- a/sql/conn_handler/socket_connection.cc
+++ b/sql/conn_handler/socket_connection.cc
@@ Channel_info *Mysqld_socket_listener::listen_for_connection_event() {
#ifdef HAVE_POLL
+ /* REPRO ONLY: widen the check-then-poll race window */
+ if (getenv("MYSQLD_REPRO_LISTENER_RACE")) sleep(10);
int retval = poll(&m_poll_info.m_fds[0], m_socket_vector.size(), -1);
#else
2. Start the server with the env var set, and make sure no clients connect:
MYSQLD_REPRO_LISTENER_RACE=1 mysqld --datadir=... &
(do not open any client connections)
3. After the server is fully up, trigger shutdown from another shell:
mysqladmin -uroot shutdown # or: kill -TERM <mysqld_pid>
Timing: the flag is set and the signal is sent while the main thread is
inside the injected 10s sleep, i.e. after the flag check but before poll().
4. Expected: mysqld exits cleanly.
Actual: mysqld never exits. Attaching gdb shows the deadlock:
gdb -p <mysqld_pid> -batch -ex 'thread apply all bt'
- main thread: poll() <- Mysqld_socket_listener::listen_for_connection_event()
- signal thread: pthread_cond_wait() <- waiting on COND_socket_listener_active
5. Proof that the signal was lost (not the flag): from a third shell, attempt
any connection to the listening port, e.g.
mysql -h127.0.0.1 -P <port> -uroot # even a failed login is enough
poll() wakes, the loop sees connection_events_loop_aborted()==true, the
listener exits, and shutdown completes immediately.
The same hang exists in 5.7 with `abort_loop` and SIGUSR1 in
handle_connections_sockets()/close_connections().
Suggested fix:
Any of the following closes the check-then-sleep window:
1. Use pselect()/ppoll() with a signal mask: keep SIGALRM blocked in the
listener thread and atomically unblock it only while inside
pselect()/ppoll(). This is the standard fix for this class of race.
2. Self-pipe/eventfd: the shutdown path writes a byte to a pipe that the
listener polls together with the listen sockets; no signal involved.
3. Cheap mitigation: give poll() a finite timeout (e.g. 1 s) so the loop
re-checks the abort flag periodically; worst case adds a bounded delay
to shutdown instead of an unbounded hang.
© 2026, Oracle Corporation and/or its affiliates
Description: The connection accept loop and the shutdown path have a classic check-then-sleep race that can make mysqld hang forever during shutdown. Accept loop (main thread), sql/conn_handler/connection_acceptor.h:64: while (!connection_events_loop_aborted()) { Channel_info *channel_info = m_listener->listen_for_connection_event(); ... } Mysqld_socket_listener::listen_for_connection_event() then blocks in an infinite-timeout poll, sql/conn_handler/socket_connection.cc:1351: int retval = poll(&m_poll_info.m_fds[0], m_socket_vector.size(), -1); Shutdown path (signal handling thread), sql/mysqld.cc:3902-3924: if (!connection_events_loop_aborted()) { set_connection_events_loop_aborted(true); ... mysql_mutex_lock(&LOCK_socket_listener_active); while (socket_listener_active) { if (pthread_kill(main_thread_id, SIGALRM)) { assert(false); break; } mysql_cond_wait(&COND_socket_listener_active, &LOCK_socket_listener_active); } mysql_mutex_unlock(&LOCK_socket_listener_active); close_connections(); } The wake-up signal is handled by an empty handler (sql/mysqld.cc:3699, installed without SA_RESTART at sql/mysqld.cc:3746-3748), whose only purpose is to interrupt poll() with EINTR. The race window: 1. Main thread evaluates `while (!connection_events_loop_aborted())` -> false (flag not set yet). 2. Before the main thread enters poll(), the shutdown thread sets the flag to true and sends the wake-up signal (SIGUSR1 in 5.7 and 8.0.0-8.0.18; SIGALRM since 8.0.19, per WL-13689). 3. The empty signal handler runs on the main thread and returns; the signal is consumed *before* poll() starts, so nothing will ever interrupt the upcoming poll(-1) call. 4. Main thread enters poll(-1). If no new connection ever arrives (typical once shutdown begins and clients have drained), poll sleeps forever. Resulting deadlock: - Main thread: blocked forever in poll(-1). - Signal thread: sends the signal exactly once, then blocks forever in mysql_cond_wait() on COND_socket_listener_active, waiting for the main thread to leave the accept loop (socket_listener_active=false is only set after the loop exits, sql/mysqld.cc:10299-10303). Shutdown never completes; mysqld must be killed with SIGKILL. The atomic abort flag (renamed from `abort_loop` and made atomic by BUG#2013467) only guarantees *visibility* of the flag; it cannot make the flag-check + poll-enter sequence atomic, so it does not close this window. The LOCK_socket_listener_active/COND handshake is only a rendezvous for the normal case (signal arrives while poll is blocked -> EINTR -> loop re-checks flag); it sends the signal only once and does not retry, so it does not mitigate a lost signal either. Any later connection attempt (even a failed TCP connect / health-check probe) wakes poll(), the loop sees the flag and exits, so in busy/ probed deployments the hang self-heals. On a quiet server the hang is permanent. The separate admin-interface listener thread (handle_admin_socket(), socket_connection.cc:1080-1082, woken by pthread_kill(admin_socket_thread_id, SIGALRM) in close_listener(), socket_connection.cc:1468) has the identical pattern and the identical race. Note: this mechanism has been unchanged since WL#7260 (2013). WL-13689 (8.0.19) only repurposed SIGUSR1 -> SIGALRM for the wake-up ("SIGALRM will replace SIGUSR1 for all its previous uses"); the race window is unaffected. How to repeat: The un-instrumented race window is only a few instructions wide, so a deterministic repro widens it with a small debug patch that simulates a scheduler preemption at exactly the wrong spot (the same thing a context switch can do in production). 1. Apply this instrumentation patch to mysqld and rebuild: --- a/sql/conn_handler/socket_connection.cc +++ b/sql/conn_handler/socket_connection.cc @@ Channel_info *Mysqld_socket_listener::listen_for_connection_event() { #ifdef HAVE_POLL + /* REPRO ONLY: widen the check-then-poll race window */ + if (getenv("MYSQLD_REPRO_LISTENER_RACE")) sleep(10); int retval = poll(&m_poll_info.m_fds[0], m_socket_vector.size(), -1); #else 2. Start the server with the env var set, and make sure no clients connect: MYSQLD_REPRO_LISTENER_RACE=1 mysqld --datadir=... & (do not open any client connections) 3. After the server is fully up, trigger shutdown from another shell: mysqladmin -uroot shutdown # or: kill -TERM <mysqld_pid> Timing: the flag is set and the signal is sent while the main thread is inside the injected 10s sleep, i.e. after the flag check but before poll(). 4. Expected: mysqld exits cleanly. Actual: mysqld never exits. Attaching gdb shows the deadlock: gdb -p <mysqld_pid> -batch -ex 'thread apply all bt' - main thread: poll() <- Mysqld_socket_listener::listen_for_connection_event() - signal thread: pthread_cond_wait() <- waiting on COND_socket_listener_active 5. Proof that the signal was lost (not the flag): from a third shell, attempt any connection to the listening port, e.g. mysql -h127.0.0.1 -P <port> -uroot # even a failed login is enough poll() wakes, the loop sees connection_events_loop_aborted()==true, the listener exits, and shutdown completes immediately. The same hang exists in 5.7 with `abort_loop` and SIGUSR1 in handle_connections_sockets()/close_connections(). Suggested fix: Any of the following closes the check-then-sleep window: 1. Use pselect()/ppoll() with a signal mask: keep SIGALRM blocked in the listener thread and atomically unblock it only while inside pselect()/ppoll(). This is the standard fix for this class of race. 2. Self-pipe/eventfd: the shutdown path writes a byte to a pipe that the listener polls together with the listen sockets; no signal involved. 3. Cheap mitigation: give poll() a finite timeout (e.g. 1 s) so the loop re-checks the abort flag periodically; worst case adds a bounded delay to shutdown instead of an unbounded hang.