MySQL Bugs: #70757: InnoDB Memcached leaks memory if innodb_api_enable

Bug #70757	InnoDB Memcached leaks memory if innodb_api_enable_binlog = 1
Submitted:	28 Oct 2013 21:34	Modified:	18 Nov 2013 14:32
Reporter:	Hartmut Holzgraefe	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Memcached	Severity:	S2 (Serious)
Version:	mysql-5.6.14	OS:	Linux
Assigned to:		CPU Architecture:	Any

Description:
The InnoDB Memcached leaks memory if innodb_api_enable_binlog = 1,
even when just updating the same single key with the same value over and over again 

with innodb_api_enable_binlog = 0 mysqld size will stay steady though

How to repeat:
Set up mysqld with innodb memcached plugin example setup according to

  http://dev.mysql.com/doc/refman/5.6/en/innodb-memcached-setup.html

and the following minimal my.cnf

  [mysqld]
  innodb_api_enable_binlog = 1        # 1 causes leak, 0 is fine
  log-bin                             # required if innodb_api_enable_binlog = 1
  innodb_flush_log_at_trx_commit = 0  # for better test speed

then just repeatedly SET the same key/value over and over again, e.g. using the following PHP snippet:

  <?php
    $loop = 0;
    $memcache_obj = memcache_connect('localhost', 11211) or die("can't connect");
    while (true) {
      memcache_set($memcache_obj, 'var_key', 'some variable', 0, 30) or die("can't set");
      $loop++;
      if ($loop % 10000 == 0) echo "Loop $loop\n";
    }

On my system this will do around 10K set commands per second and leak 1MB every ~2 seconds

Version: '5.6.14-log'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)
Dumping heap profile to mybin.hprof.0038.heap (455 MB currently in use)
Dumping heap profile to mybin.hprof.0039.heap (456 MB currently in use)
....
Dumping heap profile to mybin.hprof.0084.heap (499 MB currently in use)
Dumping heap profile to mybin.hprof.0085.heap (500 MB currently in use)
Dumping heap profile to mybin.hprof.0086.heap (501 MB currently in use)
Dumping heap profile to mybin.hprof.0087.heap (502 MB currently in use)

not sure if i read this correctly, but it's leaking the binlog filename ?

Attachment: bug70757_5.6.14_heap_profile.pdf (application/pdf, text), 8.45 KiB.

The attached pdf file shows that the leak occurs in THD::set_trans_pos(),
sql_class.h:2923

The code there looks like this:

      MEM_ROOT *log_file_mem_root= &main_mem_root;
      if (!m_trans_fixed_log_file)
        m_trans_fixed_log_file= new (log_file_mem_root) char[FN_REFLEN + 1];
      m_trans_fixed_log_file= strdup_root(log_file_mem_root,
                                          file + dirname_length(file));

this begs for question: if we have the last statement

  m_trans_fixed_log_file= strdup_root(...)

why do we need the m_trans_fixed_log_file= new (log_file_mem_root) ...  ?  
The result of new (...) will never be used, right?

Fixed in 5.6+.

Documented fix in the 5.6.16 and 5.7.4 changelogs as follows:

        Using the InnoDB memcached plugin with innodb_api_enable_binlog 
        set to 1 caused the server to leak memory.

Closed.

5.6$ bzr log -r 5632
------------------------------------------------------------
revno: 5632
committer: Libing Song <libing.song@oracle.com>
branch nick: mysql-5.6
timestamp: Sat 2013-11-16 10:06:16 +0800
message:
  Bug#17675622 INNODB MEMCACHED LEAKS MEMORY IF INNODB_API_ENABLE_BINLOG = 1
  
  The binlog name of current transaction is stored into thread buffer when the
  events are flushed into binlog. It is used by semisync later. Thread buffer
  is freed automatically after each server command(e.g COM_QUEYR). But Innodb
  memcached doesn't use server command, hence the thread buffer is not freed until
  the THD object is freed. So the thread buffer takes more and more memory after
  every binlog flush.
  
  To fix it, strcpy is used to replace strdup_root. For memcached, buffer for
  m_trans_fixed_log_file is allocated only once when it is used first time and
  following binlog operations just copy binlog name into m_trans_fixed_log_file.
  char[] cannot allocate from thread buffer by using new operator. So it is replaced
  by alloc_root.