Bug #88428 mysql_real_query hangs with EINTR errno (using YASSL)
Submitted: 9 Nov 2017 22:29 Modified: 21 Feb 2018 17:45
Reporter: Steve Brisson Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: Security: Encryption Severity:S2 (Serious)
Version:5.7.20 OS:Ubuntu
Assigned to: CPU Architecture:Any

[9 Nov 2017 22:29] Steve Brisson
Description:
OS: Ubuntu 16.04.3 LTS server
Relevant libs: libmysqlclient.so.20.3.7

I have a client program running mysql queries against a remote db and also forking to exec programs on a separate process. One of my test instances is repeatedly experiencing a hang in mysql_real_query (process consuming 100% cpu) and does not recover.

How to repeat:
I built a debug version of libmysqlclient.so.20.3.7 and found that the program is spinning in the while loop in the function net_read_raw_loop [sql/net_serv.cc]. In this state, SSL_read is failing with a SSL_get_error of 114 (YasslError receive_error) and because errno is EINTR (not sure what system call set this originally), the net_should_retry [sql/net_serv.cc] function will always return true.

This results in an infinite loop that will not ever recover.

Suggested fix:
Temporarily I have hack/patched the issue by treating yassl errors the same as SSL_ERROR_SSL in the ssl_set_sys_error [vio/viossl.c] function and also clearing errno after an EINTR is detected in vio_should_retry [vio/viosocket.c].

Looking for feedback on if there is an issue with yassl error handling in libmysqlclient or I have simply hacked around some other problem.
[10 Nov 2017 15:51] Miguel Solorzano
Thank you for the bug report. Please provide test case (C code file) attaching it using the Files tab and the command line you used to compile. Thanks in advance.
[10 Nov 2017 16:50] Steve Brisson
I don't have a good test case to reproduce the problem. The mysql client code I am using is part of a larger software package. The issue also seems very timing dependent so the size of the database is a factor as well. I was hoping code inspection would be enough to identify an issue or clear any misunderstandings I might have about how the code is supposed to work.

Again, I think my problem starts with SSL_get_error returning a 114 and this not being handled in viossl. That, combined with errno being set to EINTR creates the infinite loop in net_read_raw_loop.
[14 Nov 2017 12:04] Chiranjeevi Battula
Hello  Steve,

Thank you for the feedback.
Could you please provide repeatable test case, server logs (exact steps/sample code,  return code from server etc. - please make it as private if you prefer) to confirm this issue at our end?

Thanks,
Chiranjeevi.
[20 Nov 2017 17:09] Steve Brisson
I don't understand what you are asking for here. I don't have test code or server logs as I don't think they are relevant.

This is an issue I am having in libmysqlclient and I have given you the lines of code and errno state which is causing a hang. I believe that this could happen on any mysql client and would not be dependent on implementation.
[21 Nov 2017 14:17] Sinisa Milivojevic
Hi!

We have taken a deep lock at the problem you are describing and this is indeed a small bug in how we are using SSL library. This is a small optimisation that has number of glitches, like the one you described.

Verified as reported.
[28 Dec 2017 13:49] Sinisa Milivojevic
Hi!

We would like to inform you that we are working on this bug, but in order to proceed, we do need you to confirm the scenario in which this bug is occurring.

So, please, answer the following two questions:

* Can you confirm that the hang in mysql_real_query and that it occurs because of constant EINTR error from the operating system ???

* Also, please, let us know whether you are using YaSSL and you are not using at all OpenSSL ??? Did you try OpenSSL ????

Thank you very much in advance.
[28 Dec 2017 19:28] Steve Brisson
* Can you confirm that the hang in mysql_real_query and that it occurs because of constant EINTR error from the operating system ???

Yes, I only experienced the hang when errno was EINTR.

* Also, please, let us know whether you are using YaSSL and you are not using at all OpenSSL ??? Did you try OpenSSL ????

I am using YaSSL. I did not try OpenSSL.
[29 Dec 2017 12:31] Sinisa Milivojevic
Thank you for your feedback.
[29 Jan 2018 14:12] Sinisa Milivojevic
A patch to get a verbose output from YaSSL EINTR problems

Attachment: yassl_verbose_29-01.patch (application/octet-stream, text), 5.95 KiB.

[29 Jan 2018 14:14] Sinisa Milivojevic
Hi Mr. Brisson,

We have problem in repeating the situation that you describe. Hence, therefore, we have left in the "Files" tab a code patch. Please, apply a code patch, regenerate the problematic situation of YaSSL and when you see verbose diagnostic in your error log, please upload it to the "Files" tab.

Let us know if you have any question.
[20 Feb 2018 13:33] Sinisa Milivojevic
Hi!

Can you please try the patch that we have attached here ????

It is very important for us.
[21 Feb 2018 17:13] Steve Brisson
I'm no longer in a position where I can make test changes to the server so I am unable to try out your patch.

My own solution was to handle yassl errors in the ssl_set_sys_error function in viossl.c. In the ssl_error switch statement you simply need to set the variable 'error' appropriately on a yassl error (similar to how case SSL_ERROR_SLL is handled). I believe the yassl error range is 101 to 122.
[21 Feb 2018 17:45] Sinisa Milivojevic
Thank you .....