Bug #40662 TCP connection remains in TIME_WAIT status
Submitted: 12 Nov 2008 9:57 Modified: 20 Nov 2008 10:31
Reporter: Oli Sennhauser Email Updates:
Status: Won't fix Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:6.3.17 OS:Any
Assigned to: Magnus Blåudd CPU Architecture:Any

[12 Nov 2008 9:57] Oli Sennhauser
Description:
We have experienced that closing a connection to the mgm api will not close the TCP connection in a reasonable amount of time.

This can cause some issues if we have a high rate of connects.

How to repeat:
shell> while [ 1 ] ; do ./test_connect ; done

shell> netstat -na | grep WAIT
tcp        0      0 127.0.0.1:35258         127.0.0.1:1186 TIME_WAIT
tcp        0      0 127.0.0.1:56286         127.0.0.1:1186 TIME_WAIT
tcp        0      0 127.0.0.1:56284         127.0.0.1:1186 TIME_WAIT
tcp        0      0 127.0.0.1:35257         127.0.0.1:1186 TIME_WAIT
tcp        0      0 159.107.26.51:47944     153.88.253.11:8080 TIME_WAIT
tcp        0      0 127.0.0.1:56283         127.0.0.1:1186 TIME_WAIT
tcp        0      0 159.107.26.51:47940     153.88.253.11:8080 TIME_WAIT
shell> netstat -na | grep WAIT | wc -l
7
...
shell> netstat -na | grep WAIT | wc -l
137

Suggested fix:
Magnus S. has suggested a patch.
[12 Nov 2008 10:13] Oli Sennhauser
Makefile

Attachment: Makefile (application/octet-stream, text), 1.08 KiB.

[12 Nov 2008 10:13] Oli Sennhauser
test case

Attachment: test_connect.cpp (text/plain), 1.88 KiB.

[12 Nov 2008 14:22] Jonas Oreland
R1: no risk
E3: some work finding out why it's really needed
[20 Nov 2008 10:30] Magnus Blåudd
After researching I got this nice reply from Jan Wedvik:

A TCP connection will enter the TIME_WAIT state even if you close it
properly. The reason is this: A TCP socket is uniquely identified by the
quadruple of (IP-address A, port A, IP-address B, port B). If the
connection is closed (gracefully), and then a new connection is opened
immediately afterwards, there is a chance that a delayed packet from the
previous connection will arrive late and interfere with the new
connection. For that reason, TCP will hold the socket in the TIME_WAIT
state for a minute or so to prevent new connections using the same
IP-address/port quadruple. The OS usually allows you to configure this
interval. (The competing OSI transport protocol uses a connection number
in addition to the address/port quadruple, allowing multiple concurrent
connection between the same addresses and ports.)
A consequence of this is that if your application opens and closes
connections between a pair of hosts rapidly, you will eventually run out
of available address/port quadruples.

See also http://zeeky.mzrahman.com/archives/000019.html and
http://tangentsoft.net/wskfaq/articles/debugging-tcp.html
[20 Nov 2008 10:31] Magnus Blåudd
Will not fix since it's expected behaviour
[24 Sep 2010 6:50] Gilles Rayrat
This error looks similar to JDBC bug http://bugs.mysql.com/bug.php?id=56979
The TIME_WAIT state is expected, but on the client side, not on the server