Bug #92394 libmysqlclient enters infinite loop after signal (race condition)
Submitted: 12 Sep 2018 23:52 Modified: 28 Oct 2018 12:02
Reporter: Vincent Busam Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: C API (client library) Severity:S3 (Non-critical)
Version:5.7.23 OS:Ubuntu
Assigned to: CPU Architecture:x86

[12 Sep 2018 23:52] Vincent Busam
Description:
A process handling signals will eventually get stuck in an infinite loop in libmysqlclient.so when receiving signals at the wrong time (race condition).

Debug output:
>mysql_commit
<mysql_commit 4940
>mysql_real_query
| enter: handle: 0x547a250
| query: Query = 'commit'
| >mysql_send_query
| <mysql_send_query 5321
| >cli_advanced_command
| | >net_clear
| | <net_clear 209
| | >net_write_command
| | | enter: length: 6
| | | >net_flush
| | | | >net_write_packet
| | | | | >vio_ssl_write
| | | | | | >vio_write
| | | | | | <vio_write 214
| | | | | <vio_ssl_write 240
| | | | <net_write_packet 648
| | | <net_flush 228
| | <net_write_command 403
| | exit: result: 0
| <cli_advanced_command 1357
| >cli_read_query_result
| | >vio_ssl_read
| | | >vio_read
| | | <vio_read 136
| | | >report_errors
| | | | error: error: socket layer receive error
| | | | info: socket_errno: 4
| | | <report_errors 51
| | <vio_ssl_read 201
| | >vio_ssl_read
| | | >report_errors
| | | | error: error: socket layer receive error
| | | | info: socket_errno: 4
| | | <report_errors 51
| | <vio_ssl_read 201
| | >vio_ssl_read
| | | >report_errors
| | | | error: error: socket layer receive error
| | | | info: socket_errno: 4
| | | <report_errors 51
| | <vio_ssl_read 201
| | >vio_ssl_read
| | | >report_errors
| | | | error: error: socket layer receive error
| | | | info: socket_errno: 4
| | | <report_errors 51
| | <vio_ssl_read 201

Relevant stack trace:
#3  0x00007f7faa88d724 in vio_ssl_read (vio=0x6d58f60, buf=0x2e675d0 "\273\003", size=4) at /tmp/mysql-5.7-5.7.23/vio/viossl.c:172
#4  0x00007f7faa85fe59 in net_read_raw_loop (net=0x4d23550, count=4) at /tmp/mysql-5.7-5.7.23/sql/net_serv.cc:672

In vio_ssl_read  at /tmp/mysql-5.7-5.7.23/vio/viossl.c:167
SSL_read() always returns -1
ssl_get_error() returns 114

How to repeat:
Execute this python script, then send it a LOT of TERM signals.  Eventually it'll lock up using 100% CPU.

#!/usr/bin/python2 -u
import signal
import MySQLdb

v = False

def handler(a, b):
    global v
    v = True

signal.signal(signal.SIGTERM, handler)

db = MySQLdb.connect(.....)
cursor = db.cursor()
while True:
    cnt = cursor.execute("SELECT id FROM my_table")
    print cnt
    for row in cursor.fetchall():
        pass

Suggested fix:
It appears the SSL errors aren't properly cleared before retrying.
[27 Sep 2018 13:29] Sinisa Milivojevic
Hi,

First of all, are you using our package or somebody else's package or are you building our binaries yourself.

If you are not using our package, please use it and see if you can repeat the problem.

Same goes for the Python driver for MySQL. Use Connecter/Python instead.

Most important of all, the two problematic functions are not part of our code, but a part of your OS' system libraries. We are not maintaining those but are simply finding bugs to them, once that we find them. Our documentation that comes with 5.7.23 package clearly states which SSL library to use and which version.
[27 Sep 2018 21:01] Vincent Busam
I originally found this using the stock Ubuntu package.  I re-verified it with the MySQL source (built with debug symbols to get a useful traceback).  The lockup is entirely in the vio/yassl code included with the MySQL source distribution.

For a work-around we are blocking signals while entering into libmysqlclient calls.
[28 Sep 2018 12:02] Sinisa Milivojevic
Hi,

Once again, please download and try our server and client binaries that you can download from our site. Also use our Connector/Python.

For your information, we have switched to OpenSSL.
[29 Oct 2018 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".