Bug #75259 Exception when reading very large responses under Python 3
Submitted: 18 Dec 2014 11:54 Modified: 17 Jul 2015 8:15
Reporter: J. Stohner Email Updates:
Status: Won't fix Impact on me:
None 
Category:Connector / Python Severity:S2 (Serious)
Version:1.2.3 OS:Any
Assigned to: CPU Architecture:Any

[18 Dec 2014 11:54] J. Stohner
Description:
There is a bug in line 318 of ./python3/mysql/connector/protocol.py within the method read_text_result(). If the response indicates a large value (if packet.startwith(b'\xff\xff\xff'), the single packets (containing bytes) are collected in the list 'datas'. When the last packet is received, the rowdata is built by using ''.join(). The join function now tries to join the byte values with the '' string, which fails and an exception is raised.

[...]
  File "/home/user/mysql-connector-python-1.2.3/build/testing/mysql/connector/protocol.py", line 318, in read_text_result
    rowdata = utils.read_lc_string_list(''.join(datas))
TypeError: sequence item 0: expected str instance, bytes found

How to repeat:
The following test (added to test_bugs.py) can repeat the error:

class BugJSTests(tests.MySQLConnectorTests):

    def test_execute_return(self):
        """Connector fails to read very large values (e.g. string of 100MB size)"""

        config = tests.get_mysql_config()
        # We are writing and reading 100MB, which will take reasonable                                                                                                    
        # time depending on your system, so set a large timeout.                                                                                                          
        config['connection_timeout'] = 100
        cnx = connection.MySQLConnection(**config)
        cur = cnx.cursor()
        tbl = "bugjstest"
        # This is the value, set at 100MB of 'A's                                                                                                                         
        value = "A"*104857600
        # We have to check if max_allowed_packet is sufficiently long                                                                                                     
        cur.execute("SELECT @@max_allowed_packet")
        data = cur.fetchall()
        self.assertGreater(data[0][0], len(value), "max_allowed_packet is to small to execute this test!")
        cur.execute("DROP TABLE IF EXISTS %s" % tbl)
        cur.execute("CREATE TABLE %s (id LONGTEXT)" % tbl)
        cur.execute("INSERT INTO %s VALUES (%%s)" % tbl, (value, ))
        self.assertEqual(1, cur.rowcount)
        res = cur.execute("SELECT * from %s" % tbl)
        data = cur.fetchall()
        # We only check the numer of rows and the *length* of the value                                                                                                   
        # so that the logfile is not polluted with the large value.                                                                                                       
        # Comparing the value does not matter for the test because                                                                                                        
        # the buggy versions of Connector will fail to build the value                                                                                                    
        # and an exception will be raised.                                                                                                                                
        self.assertEqual(1, len(data))
        self.assertEqual(len(value), len(data[0][0]))
        cur.close()
        cnx.close()

Note that max_allowed_packet has to be increased to make the test run. The only way that worked for me to make the test run was the following change in ./tests/mysqld.py:

dtsp@dtsp-dev:~/mysql-connector-python-1.2.3/tests> diff -c3  mysqld.py~ mysqld.py
*** mysqld.py~  2014-12-18 10:10:54.566079419 +0100
--- mysqld.py   2014-12-18 11:00:16.776465044 +0100
***************
*** 214,219 ****
--- 214,220 ----
          cmd = [
              os.path.join(self._sbindir, EXEC_MYSQLD),
              "--defaults-file={0}".format(self._option_file),
+             "--max_allowed_packet=800000000",
          ]
  
          if os.name == 'nt':

Suggested fix:
The fix is easy, just add a 'b' prefix to the empty string in line 318 of ./python3/mysql/connector/protocol.py:

root# diff -c3 protocol.py*
*** protocol.py 2014-10-26 23:12:01.700401176 +0000
--- protocol.py.orig    2014-08-14 21:47:44.000000000 +0000
***************
*** 315,321 ****
                      eof = self.parse_eof(packet)
                  else:
                      datas.append(packet[4:])
!                 rowdata = utils.read_lc_string_list(b''.join(datas))
              elif packet[4] == 254:
                  eof = self.parse_eof(packet)
                  rowdata = None
--- 315,321 ----
                      eof = self.parse_eof(packet)
                  else:
                      datas.append(packet[4:])
!                 rowdata = utils.read_lc_string_list(''.join(datas))
              elif packet[4] == 254:
                  eof = self.parse_eof(packet)
                  rowdata = None
[18 Dec 2014 12:08] J. Stohner
Connector/Python 2.0.2 is not affected, refer to line 320 of ./lib/mysql/connector/protocol.py.
[5 Jan 2015 11:33] Geert Vanderkelen
Thanks for this bug report and finding the fix.

Is there a need for this to be fixed in v1.2; can you use v2.0 of Connector/Python instead?
[14 Jan 2015 12:36] J. Stohner
I can use version 2 for my purposes. It looks like that you are preparing an update to 1.2 (1.2.4 "not yet released"), so I thought the bug report would be helpful.
[17 Jul 2015 8:15] Andrii Nikitin
No releases are planned for 1.* and since 2.0 is not affected - I close this bug with 'Won't fix' resolution.

If somebody needs it to be fixed in 1.* - please state your reasoning and it will be reviewed if anything can be done about that