Bug #87355 innodb_memcache OOM when reading >8k values
Submitted: 9 Aug 2017 9:23 Modified: 10 Aug 2017 7:48
Reporter: Nikolai Ikhalainen Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: Memcached Severity:S3 (Non-critical)
Version:5.7.17, 5.7.19 OS:Any
Assigned to: CPU Architecture:Any

[9 Aug 2017 9:23] Nikolai Ikhalainen
Description:
Original bug description: https://bugs.launchpad.net/percona-server/+bug/1655403
includes quick fix.

setup innodb memcache, create a table with 65k varchar field.
Create single connection to memcache port.
set/get in a loop without closing connection
at some moment oom kills mysqld
#!/usr/bin/env ruby

    require 'socket'

    def bin_to_hex(s)
      s.each_byte.map { |b| b.to_s(16) }.join
    end

    class MemcacheClient
      def initialize
        @s = TCPSocket.new 'localhost', 11211
      end

      def get(key)
        @s.write("get #{key}\r\n")
        info = @s.gets
        if info.start_with?('VALUE')
          count = info.split[-1].to_i
          value = @s.read(count)
          value += @s.gets # \r\n
          value += @s.gets # END
        end
        info + (value || "")
      end

      def set(key, value)
        value = value.to_s
        @s.write("set #{key} 0 0 #{value.bytesize}\r\n#{value}\r\n")
        @s.gets
      end
    end

    def test_write_and_read(size=64000)
      memcache_client = MemcacheClient.new
      random = Random.new

      1.upto(1_000_000) do |n|
        k = bin_to_hex(random.bytes(20))
        v = 'a' * size

        puts n
        $stdout.flush

        memcache_client.set(k, v)
        ret = memcache_client.get(k)
        if !ret.include?(v)
          raise "Error: mismatch: #{ret.inspect} != #{v.inspect}"
        end
      end
    end
    test_write_and_read

How to repeat:
docker run -it --rm --name m57 -e MYSQL_ALLOW_EMPTY_PASSWORD=1 mysql/mysql-server:5.7
inside container:
yum install ruby
cat /usr/share/mysql/innodb_memcached_config.sql | sed -e 's/c2 VARCHAR(1024)/c2 VARCHAR(65000)/' > innodb_memcached_config_changed.sql

mysql < innodb_memcached_config_changed.sql
mysql -e 'INSTALL PLUGIN daemon_memcached soname "libmemcached.so";'

ruby test_memcache.rb

Monitor memory consumption top -p $( pgrep -xn mysqld )
when script prints 75k  5GB RAM consumed.
If script killed memory consumption returns back to 300M
[10 Aug 2017 7:48] MySQL Verification Team
Hello Nikolai,

Thank you for the report and test case.

Thanks,
Umesh
[10 Aug 2017 7:49] MySQL Verification Team
test results - 5.7.19

Attachment: 87355.results (application/octet-stream, text), 4.42 KiB.