Description:
There is a bug in line 318 of ./python3/mysql/connector/protocol.py within the method read_text_result(). If the response indicates a large value (if packet.startwith(b'\xff\xff\xff'), the single packets (containing bytes) are collected in the list 'datas'. When the last packet is received, the rowdata is built by using ''.join(). The join function now tries to join the byte values with the '' string, which fails and an exception is raised.
[...]
File "/home/user/mysql-connector-python-1.2.3/build/testing/mysql/connector/protocol.py", line 318, in read_text_result
rowdata = utils.read_lc_string_list(''.join(datas))
TypeError: sequence item 0: expected str instance, bytes found
How to repeat:
The following test (added to test_bugs.py) can repeat the error:
class BugJSTests(tests.MySQLConnectorTests):
def test_execute_return(self):
"""Connector fails to read very large values (e.g. string of 100MB size)"""
config = tests.get_mysql_config()
# We are writing and reading 100MB, which will take reasonable
# time depending on your system, so set a large timeout.
config['connection_timeout'] = 100
cnx = connection.MySQLConnection(**config)
cur = cnx.cursor()
tbl = "bugjstest"
# This is the value, set at 100MB of 'A's
value = "A"*104857600
# We have to check if max_allowed_packet is sufficiently long
cur.execute("SELECT @@max_allowed_packet")
data = cur.fetchall()
self.assertGreater(data[0][0], len(value), "max_allowed_packet is to small to execute this test!")
cur.execute("DROP TABLE IF EXISTS %s" % tbl)
cur.execute("CREATE TABLE %s (id LONGTEXT)" % tbl)
cur.execute("INSERT INTO %s VALUES (%%s)" % tbl, (value, ))
self.assertEqual(1, cur.rowcount)
res = cur.execute("SELECT * from %s" % tbl)
data = cur.fetchall()
# We only check the numer of rows and the *length* of the value
# so that the logfile is not polluted with the large value.
# Comparing the value does not matter for the test because
# the buggy versions of Connector will fail to build the value
# and an exception will be raised.
self.assertEqual(1, len(data))
self.assertEqual(len(value), len(data[0][0]))
cur.close()
cnx.close()
Note that max_allowed_packet has to be increased to make the test run. The only way that worked for me to make the test run was the following change in ./tests/mysqld.py:
dtsp@dtsp-dev:~/mysql-connector-python-1.2.3/tests> diff -c3 mysqld.py~ mysqld.py
*** mysqld.py~ 2014-12-18 10:10:54.566079419 +0100
--- mysqld.py 2014-12-18 11:00:16.776465044 +0100
***************
*** 214,219 ****
--- 214,220 ----
cmd = [
os.path.join(self._sbindir, EXEC_MYSQLD),
"--defaults-file={0}".format(self._option_file),
+ "--max_allowed_packet=800000000",
]
if os.name == 'nt':
Suggested fix:
The fix is easy, just add a 'b' prefix to the empty string in line 318 of ./python3/mysql/connector/protocol.py:
root# diff -c3 protocol.py*
*** protocol.py 2014-10-26 23:12:01.700401176 +0000
--- protocol.py.orig 2014-08-14 21:47:44.000000000 +0000
***************
*** 315,321 ****
eof = self.parse_eof(packet)
else:
datas.append(packet[4:])
! rowdata = utils.read_lc_string_list(b''.join(datas))
elif packet[4] == 254:
eof = self.parse_eof(packet)
rowdata = None
--- 315,321 ----
eof = self.parse_eof(packet)
else:
datas.append(packet[4:])
! rowdata = utils.read_lc_string_list(''.join(datas))
elif packet[4] == 254:
eof = self.parse_eof(packet)
rowdata = None