Bug #82019 Is client library supposed to retry EINTR indefinitely or not
Submitted: 27 Jun 2016 14:04 Modified: 12 Aug 2016 17:12
Reporter: Laurynas Biveinis (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Connection Handling Severity:S3 (Non-critical)
Version:5.5, 5.6, 5.7 OS:Any
Assigned to:
Tags: EINTR, libmysql

[27 Jun 2016 14:04] Laurynas Biveinis
Description:
It is not clear from the source code reading what's the intended behavior for EINTR handling in libmysqlclient, and whether the intended behavior and actual implementation agree. Here's the background:

1) Up to 5.1 inclusive, client library retries EINTR indefinitely if --enable-thread-safe-client was given (which was the default):

static ulong
my_real_read(NET *net, size_t *complen)
...
#if defined(THREAD_SAFE_CLIENT) && !defined(MYSQL_SERVER)
	  if (vio_errno(net->vio) == SOCKET_EINTR)
	  {
	    DBUG_PRINT("warning",("Interrupted read. Retrying..."));
	    continue;
	  }
#endif

THREAD_SAFE_CLIENT is set by Autoconf.

2) 5.5 removed the threading-related preprocessor symbols in [1], but failed to remove the checks in the source code, effectively disabling the code above.

3) bug 65956 was logged for 5.5 stating the same (EINTR is not retried indefinitely anymore in the client), and there are interesting comments there: 

3.1) "[25 Jul 2012 14:42] Davi Arnaut Also, this behavior was changed in MySQL 5.6 and up. For the client library, interrupted I/O operations are now always retried."

But 5.6 appears to contain exactly the same EINTR logic as 5.5, only refactored, complete with the same never-defined THREAD_SAFE_CLIENT.

3.2) "[10 Jan 2013 11:07] Erlend Dahl (...) 5.6 has an implementation where read is done indefinitely in case of socket interrupts(func: net_should_retry in net_serv.cc). " 

Again, no indefinite retry logic in that function unless one assumes that THREAD_SAFE_CLIENT is actually defined.

3.3) "[16 Jan 2013 6:50] xiaobin lin (...) Davi , you are right, in 5.6, SA_RESTART makes the system call "recv " locks like blocking."

But nothing in the 5.6 source tree appears to ever set SA_RESTART, and if it did, it would appear that it would replace EINTR handling only partially.

3.4) in the same comment, "But in 5.5,  defined(THREAD_SAFE_CLIENT) is *ALWARYS* false, so error comes out.", which nobody replied to.

4) Finally, 5.7 removed the never-defined THREAD_SAFE_CLIENT symbol in what appears to be a mechanical cleanup [2].

So, is the client library supposed to retry EINTR indefinitely or not? If yes, does it actually do so, in 5.5/5.6/5.7?

[1]:

commit ebd24626ca38e7fa1e3da2acdcf88540be70fabe
Author: Magne Mahre <magne.mahre@oracle.com>
Date:   Tue Jan 11 10:07:37 2011 +0100

    Remove configuration preprocessor symbols 'THREAD'
    and 'THREAD_SAFE_CLIENT'.
    
    As of MySQL 5.5, we no longer support non-threaded
    builds.   This patch removes all references to the
    obsolete THREAD and THREAD_SAFE_CLIENT preprocessor
    symbols.  These were used to distinguish between
    threaded and non-threaded builds.

[2]:

commit de956a045830da4bd68cc870aa42c35177fdad4a
Author: Jon Olav Hauglid <jon.hauglid@oracle.com>
Date:   Fri May 24 15:50:06 2013 +0200

    Bug#16792349: YET MORE UNUSED #IFDEFS
    
    Additional patch, remove the following unused #ifdefs:
    DONT_ALLOW_UNSIGNED_PLUS
    IN_THE_FUTURE
    LIKE_CMP_TOUPPER
    TO_BE_REMOVED
    SECURE_SOCKETS
    THREAD_SPECIFIC_SIGPIPE
    IGNORE_SIGHUP_SIGQUIT
    THREAD_SAFE_CLIENT
    SHOULD_BE_CHECKED
    DONT_ALLOW_FULL_LOAD_DATA_PATHS
    TO_BE_IMPLEMENTED
    ALLOW_LINESEPARATOR_IN_STRINGS
    NOT_NEEDED
    ERROR_ON_NO_SIZEOF_PLUGIN_SYMBOL
    FIX_LATER
    RESTRICTED_GROUP
    QQ_ALL_HANDLERS_SUPPORT_VARCHAR
    TESTTIME
    WE_WANT_TO_SUPPORT_VERY_OLD_FRM_FILES
    TO_BE_DELETED_ON_PRODUCTION
    WE_WANT_TO_HANDLE_UNORMALIZED_DATES

How to repeat:
See above

Suggested fix:
Please clarify what's intended here, and if the actual implementation differs, please fix it.
[27 Jun 2016 17:06] Sinisa Milivojevic
Hi Laurynas,

Your analysis is quite correct. Let me introduce you to the history of this part of code. Some early Linux kernels, most of all Red Hat ones, did send a lot of EINTR which were to be ignored.

Situation is fixed now and OS function `recv` is called in the VIO layer.

This, however requires to  be documented properly.
[27 Jun 2016 19:19] Laurynas Biveinis
One thing I have seen was that running at least one client program (5.5 mysqlimport) with AddressSanitizer apparently makes it observe enough EINTRs to abort early
[28 Jun 2016 4:54] Shane Bester
perhaps this is interesting too:
http://bugs.mysql.com/bug.php?id=81258
(please document the mysql_options() MYSQL_OPT_RETRY_COUNT)
[28 Jun 2016 14:07] Sinisa Milivojevic
This should be all fixed in 5.7.
[11 Jul 2016 14:05] Larry Adams
Can someone be specific on what versions of the various clients this is fixed in?  Is it MySQL 5.5-XX, MySQL 5.6-XX, MySQL 5.7-XX?  I'm using version 5.7-12-1, and do not see either the references to MYSQL_OPT_RETRY_COUNT, or the blocking of EINTR calls, and the workaround is ..... horrible.
[12 Jul 2016 13:55] Sinisa Milivojevic
Hi,

Due to the series of unfortunate mishaps, this bug has yet to be fixed in 5.5, 5.6 and 5.7.

You will be informed on this page when it is achieved.
[15 Jul 2016 13:22] Laurynas Biveinis
Bug 82019 fix for 5.5

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: bug82019-5.5.patch (application/octet-stream, text), 1.06 KiB.

[15 Jul 2016 13:23] Laurynas Biveinis
Bug 82019 fix for 5.6

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: bug82019-5.6.patch (application/octet-stream, text), 898 bytes.

[15 Jul 2016 13:23] Laurynas Biveinis
Bug 82019 fix for 5.7

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: bug82019-5.7.patch (application/octet-stream, text), 893 bytes.

[15 Jul 2016 13:40] Larry Adams
Thanks for that.  Can Oracle provide a release schedule for these updates?

TheWitness
[15 Jul 2016 14:08] Sinisa Milivojevic
Laurynas,

Thank you for your contributions, which are welcome although we made lots of progress already.

When the changes will be pushed , this bug report will be closed by Development , with the info of the releases where the patches are pushed into.
[12 Aug 2016 17:12] Paul Dubois
Posted by developer:
 
Noted in 5.5.52, 5.6.34, 5.7.15 changelogs.

EINTR handling in the client library has been fixed so that
interrupted read and write calls are retried. Previously, EINTR was
ignored.
[12 Aug 2016 17:17] Paul Dubois
5.6.33, not 5.6.34.
[12 Aug 2016 18:16] Larry Adams
Thanks everyone for getting this critical bug addressed!
[31 Aug 2016 12:51] Ståle Deraas
Thanks Laurynas for the contributions. We changed them a bit.
[22 Sep 2016 21:18] Larry Adams
I just tested using 5.7.15, and this is NOT fixed in the client library.  Darn.  I'm still receiving EINTR errors for various client calls.  I can reproduce 100% of the time.  I someone in Oracle would like a test case, you can reach out via my registered email.

Can we please re-open this ticket?

TheWitness
[22 Sep 2016 22:14] Larry Adams
Adding additional information.  The error seems to be taking place in the mysql_real_connect() call only at the moment.  The other API calls are behaving better now.  Looking into that code, is it also using the vio subsystem?
[22 Sep 2016 22:43] Roel Van de Paar
Larry, your best bet is to always open a new bug. Re-opening a bug is highly unlikely to happen.
[22 Sep 2016 22:43] Roel Van de Paar
If you do, please add the URL here for tracking.
[22 Sep 2016 22:46] Larry Adams
Sure.  I'm digging into the code right now.  Very little documentation.  At some point I'll either have an aha moment, or go eat dinner.  Either way, I'll open a new bug and link.
[22 Sep 2016 23:20] Larry Adams
Here is a link to the new bug.  Suspecting somewhere in the low level stuff net_serv.c still.  But no idea where.

http://bugs.mysql.com/bug.php?id=83109