Bug #47571 idle named pipe connection is unkillable

• Submitted: 23 Sep 2009 22:54
• Modified: 12 Mar 2010 17:28
• Reporter: Vladislav Vaintroub
• Status: Closed
• Category: MySQL Server
• Severity: S1 (Critical)
• Version: all
• OS: Windows (Win2008 R2)
• Assigned to: Vladislav Vaintroub
• CPU Architecture: Any

[23 Sep 2009 22:54] Vladislav Vaintroub

Description:
An idle named pipe connection can not be killed with "kill" statement.

How to repeat:
Enable named pipe connections with --enable-named-pipe --socket=MySQL
start one mysql client with 
mysql --uroot --protocol=pipe

start another mysql client and issue
kill 1;
(assuming the named pipe connection id is 1)

Kill statement would hang.

After aborting mysql with CRTL-C(twice), I start another one and issue "show processlist". It hangs.

Suggested fix:
Debugging shows, that mysqld hangs at an attempt to Close pipe handle with 
CloseHandle while there is another thread that stucks in ReadFile initiated by vio_read_pipe.

When pipe is used synchronously, all operations on it are serialized, which means CloseHandle would not interrupt a running read.

I suggest to open the named pipe in async mode (CreateFile with FILE_FLAG_OVERLAPPED). We could use event mechanism available with async IO to break the read from another thread.
We could then do something similar to shared memory read implementation, that employs WaitForMultipleObjects to detect closed connections and to restrict IO time according to timeouts.

[24 Sep 2009 12:20] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/84481

[24 Sep 2009 12:24] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/84482

[24 Sep 2009 14:03] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/84508

2814 Vladislav Vaintroub	2009-09-24
      Bug#47571: idle named pipe connection is unkillable
      Bug#31621: Windows server hanging during shutdown using named pipes 
                 and idle connection
      
      Problem: when idle pipe connection is forcefully closed with KILL
      statement or when the server goes down, thread that is closing connection
      would hang infinitely in CloseHandle(). The reason for the hang is that 
      named pipe operations are performed synchronously. In this mode all IOs
      on pipe are serialized, that is CloseHandle() will not abort ReadFile() 
      in another thread, but wait for ReadFile() to complete.
      
      The fix implements asynchrnous mode for named pipes, where operation of file
      are not synchronized. Read/Write operation would fire an async IO and wait for
      either IO completion or timeout.
      
      Note, that with this patch timeouts are properly handled for named pipes.
     @ include/violite.h
        Add pipe_overlapped to Vio structure for async IO for named pipes.
     @ sql-common/client.c
        Use async IO for named pipes.
     @ sql/mysqld.cc
        Use asynchronous pipe IO.
     @ vio/vio.c
        pass NET structure to vio_new_win32pipe 
        (it contains timeout values that are used in
        vio_read_pipe/vio_write_pipe)
     @ vio/viosocket.c
        Use async IO for named pipes.
        After issuing IO, wait for either IO completion, pipe_close_event
        or timeout.

[24 Sep 2009 14:08] Vladislav Vaintroub

difference between the last patch and previous is that I removed pipe_close_event. IT is not really necessary, if pipe is in async mode, OS will not serialize IO operations, so CloseHandle() can run in paralell with ReadFile()

[29 Oct 2009 17:10] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/88636

2829 Vladislav Vaintroub	2009-10-29
      Bug#47571: idle named pipe connection is unkillable
      Bug#31621: Windows server hanging during shutdown using named pipes 
                       and idle connection
            
      Problem: when idle pipe connection is forcefully closed with KILL
      statement or when the server goes down, thread that is closing connection
      would hang infinitely in CloseHandle(). The reason for the hang is that 
      named pipe operations are performed synchronously. In this mode all IOs
      on pipe are serialized, that is CloseHandle() will not abort ReadFile() 
      in another thread, but wait for ReadFile() to complete.
            
      The fix implements asynchrnous mode for named pipes, where operation of file
      are not synchronized. Read/Write operation would fire an async IO and wait for
      either IO completion or timeout.
            
      Note, that with this patch timeouts are properly handled for named pipes.
     @ include/violite.h
        Add pipe_overlapped to Vio structure for async IO for named pipes.
     @ sql-common/client.c
        Use asynchronous pipe IO.
     @ sql/mysqld.cc
        Use asynchronous pipe IO.
     @ vio/vio.c
        pass NET structure to vio_new_win32pipe 
        (it contains timeout values that are used in
        vio_read_pipe/vio_write_pipe)
     @ vio/viosocket.c
        Use async IO for named pipes.
        After issuing IO, wait for either IO completion, pipe_close_event
        or timeout.

[30 Oct 2009 19:57] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/88803

2829 Vladislav Vaintroub	2009-10-30
      Bug#47571: idle named pipe connection is unkillable
      Bug#31621: Windows server hanging during shutdown using named pipes 
                       and idle connection
            
      Problem: when idle pipe connection is forcefully closed with KILL
      statement or when the server goes down, thread that is closing connection
      would hang infinitely in CloseHandle(). The reason for the hang is that 
      named pipe operations are performed synchronously. In this mode all IOs
      on pipe are serialized, that is CloseHandle() will not abort ReadFile() 
      in another thread, but wait for ReadFile() to complete.
            
      The fix implements asynchrnous mode for named pipes, where operation of file
      are not synchronized. Read/Write operation would fire an async IO and wait for
      either IO completion or timeout.
            
      Note, that with this patch timeouts are properly handled for named pipes.
      
      Post-review: Win32 timeout code has been fixed for named pipes and shared
      memory. We do not store pointer to NET in vio structure, only the read and 
      write timeouts.
     @ include/violite.h
        Add pipe_overlapped to Vio structure for async IO for named pipes.
     @ sql-common/client.c
        Use asynchronous pipe IO.
     @ sql/mysqld.cc
        Use asynchronous pipe IO.
     @ vio/vio.c
        -Refactor timeouts for win32 protocols: shared memory and named pipes.
        Store read/write timeout in VIO structure, instead of storing pointer
        to NET. New function vio_win32_timeout called indirectly via 
        vio_timeout changes these values.
        
        -Initialize Overlapped for use in named pipe async IO.
     @ vio/vio_priv.h
        Remove vio_ignore_timeout.
        Add vio_win32_timeout to be used for named pipes and shared memory.
     @ vio/viosocket.c
        Use async IO for named pipes.
        After issuing IO, wait for either IO completion, pipe_close_event
        or timeout.
        
        Refactor timeouts for named pipe and shared memory.

[2 Nov 2009 22:20] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/89010

3156 Vladislav Vaintroub	2009-11-02
      Bug#47571: idle named pipe connection is unkillable
      Bug#31621: Windows server hanging during shutdown using named pipes 
                 and idle connection
                  
      Problem: when idle pipe connection is forcefully closed with KILL
      statement or when the server goes down, thread that is closing connection
      would hang infinitely in CloseHandle(). The reason for the hang is that 
      named pipe operations are performed synchronously. In this mode all IOs
      on pipe are serialized, that is CloseHandle() will not abort ReadFile() 
      in another thread, but wait for ReadFile() to complete.
                  
      The fix implements asynchrnous mode for named pipes, where operation of file
      are not synchronized. Read/Write operation would fire an async IO and wait for
      either IO completion or timeout.
                  
      Note, that with this patch timeouts are properly handled for named pipes.
            
      Post-review: Win32 timeout code has been fixed for named pipes and shared
      memory. We do not store pointer to NET in vio structure, only the read and 
      write timeouts.
     @ include/violite.h
        Add pipe_overlapped to Vio structure for async IO for named pipes.
     @ sql-common/client.c
        Use asynchronous pipe IO.
     @ sql/mysqld.cc
        Use asynchronous pipe IO.
     @ vio/vio.c
        -Refactor timeouts for win32 protocols: shared memory and named pipes.
        Store read/write timeout in VIO structure, instead of storing pointer
        to NET. New function vio_win32_timeout called indirectly via 
        vio_timeout changes these values.
     @ vio/vio_priv.h
        Remove vio_ignore_timeout.
        Add vio_win32_timeout to be used for named pipes and shared memory.
     @ vio/viosocket.c
        Use async IO for named pipes.
        After issuing IO, wait for either IO completion, pipe_close_event
        or timeout.
                
        Refactor timeouts for named pipe and shared memory.

[4 Nov 2009 9:25] Bugs System

Pushed into 5.1.41 (revid:[e-mail]-20091104092152-qz96bzlf2o1japwc) (version source revid:[e-mail]-20091103162305-08l4gkeuif2ozsoj) (merge vers: 5.1.41) (pib:13)

[6 Nov 2009 1:31] Paul DuBois

Noted in 5.1.41 changelog.

On WIndows, when an idle named pipe connection was forcibly closed
with a KILL statement or because the server was being shut down, the
thread that was closing the connection would hang infinitely.

Setting report to NDI pending push to 5.5.x.

[11 Nov 2009 6:51] Bugs System

Pushed into 6.0.14-alpha (revid:[e-mail]-20091110093407-rw5g8dys2baqkt67) (version source revid:[e-mail]-20091109080109-7dxapd5y5pxlu08w) (merge vers: 6.0.14-alpha) (pib:13)

[11 Nov 2009 7:01] Bugs System

Pushed into 5.5.0-beta (revid:[e-mail]-20091109115615-nuohp02h8mdrz8m2) (version source revid:[e-mail]-20091105091401-4fwfzjh1rldc6esy) (merge vers: 5.5.0-beta) (pib:13)

[11 Nov 2009 14:32] Paul DuBois

Noted in 5.5.0, 6.0.14 changelogs.

[20 Nov 2009 12:10] Bugs System

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/91062

3208 Vladislav Vaintroub	2009-11-20
      Bug#47571 : idle named pipe connection is unkillable
      implement Davi's review suggestions (post-push fixes)
     @ include/violite.h
        Use official abbreviation for milliseconds (ms)
     @ sql/mysqld.cc
        Fix formatting
        Add error handling for the case of CreateEvent error
     @ vio/vio.c
        Use official abbreviation for milliseconds(ms)
        Remove superfluous memset
        Fix formatting
     @ vio/viosocket.c
        Use official abbreviation for milliseconds (ms)
        Use size_t  datatype instead of int in pipe_complete_io

[2 Dec 2009 8:03] Bugs System

Pushed into 5.1.42 (revid:[e-mail]-20091202080033-mndu4sxwx19lz2zs) (version source revid:[e-mail]-20091125130912-d7hrln14ef7y5d7i) (merge vers: 5.1.42) (pib:13)

[16 Dec 2009 8:41] Bugs System

Pushed into 6.0.14-alpha (revid:[e-mail]-20091216083311-xorsasf5kopjxshf) (version source revid:[e-mail]-20091214191830-wznm8245ku8xo702) (merge vers: 6.0.14-alpha) (pib:14)

[16 Dec 2009 8:47] Bugs System

Pushed into 5.5.0-beta (revid:[e-mail]-20091216082430-s0gtzibcgkv4pqul) (version source revid:[e-mail]-20091124083136-iqm136jm31sfdwg3) (merge vers: 5.5.0-beta) (pib:14)

[16 Dec 2009 8:55] Bugs System

Pushed into mysql-next-mr (revid:[e-mail]-20091216083231-rp8ecpnvkkbhtb27) (version source revid:[e-mail]-20091212203859-fx4rx5uab47wwuzd) (merge vers: 5.6.0-beta) (pib:14)

[18 Dec 2009 10:38] Bugs System

Pushed into 5.1.41-ndb-7.1.0 (revid:[e-mail]-20091218102229-64tk47xonu3dv6r6) (version source revid:[e-mail]-20091218095730-26gwjidfsdw45dto) (merge vers: 5.1.41-ndb-7.1.0) (pib:15)

[18 Dec 2009 10:54] Bugs System

Pushed into 5.1.41-ndb-6.2.19 (revid:[e-mail]-20091218100224-vtzr0fahhsuhjsmt) (version source revid:[e-mail]-20091217101452-qwzyaig50w74xmye) (merge vers: 5.1.41-ndb-6.2.19) (pib:15)

[18 Dec 2009 11:09] Bugs System

Pushed into 5.1.41-ndb-6.3.31 (revid:[e-mail]-20091218100616-75d9tek96o6ob6k0) (version source revid:[e-mail]-20091217154335-290no45qdins5bwo) (merge vers: 5.1.41-ndb-6.3.31) (pib:15)

[18 Dec 2009 11:23] Bugs System

Pushed into 5.1.41-ndb-7.0.11 (revid:[e-mail]-20091218101303-ga32mrnr15jsa606) (version source revid:[e-mail]-20091218064304-ezreonykd9f4kelk) (merge vers: 5.1.41-ndb-7.0.11) (pib:15)

[12 Mar 2010 14:18] Bugs System

Pushed into 5.1.44-ndb-7.0.14 (revid:[e-mail]-20100312135944-t0z8s1da2orvl66x) (version source revid:[e-mail]-20100312115609-woou0te4a6s4ae9y) (merge vers: 5.1.44-ndb-7.0.14) (pib:16)

[12 Mar 2010 14:33] Bugs System

Pushed into 5.1.44-ndb-6.2.19 (revid:[e-mail]-20100312134846-tuqhd9w3tv4xgl3d) (version source revid:[e-mail]-20100312060623-mx6407w2vx76h3by) (merge vers: 5.1.44-ndb-6.2.19) (pib:16)

[12 Mar 2010 14:50] Bugs System

Pushed into 5.1.44-ndb-6.3.33 (revid:[e-mail]-20100312135724-xcw8vw2lu3mijrhn) (version source revid:[e-mail]-20100312103652-snkltsd197l7q2yg) (merge vers: 5.1.44-ndb-6.3.33) (pib:16)

[13 Jan 2011 16:40] Paul DuBois

Addition to changelog entry:

As a result of the work done for this bug, the net_read_timeout and
net_write_timeout system variables now apply to connections over all
transports, not just to TCP/IP.