Bug #13078 mysqld memleak for ndb
Submitted: 9 Sep 2005 0:55 Modified: 15 Dec 2005 13:30
Reporter: Howard Hart Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S1 (Critical)
Version:4.1,5.0 OS:Linux (Linux 2.6.12)
Assigned to: Jonas Oreland CPU Architecture:Any

[9 Sep 2005 0:55] Howard Hart 
Description:
mysqld and/or ndbd appears to use up all available memory and causes Linux to kill all mysqld processes. Under high load, mysqld/ndbd only seems to last 1 and 1/2 weeks. Initial startup shows 7 GB out of 8 GB total RAM available. Mysql logs show no activity until startup and recovery and no core dumps since Linux is killing the process:

Thu Sep  8 00:36:08 GMT 2005
050908  0:36:09  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
050908  0:36:10  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 443 1324810155.
InnoDB: Doing recovery: scanned up to log sequence number 443 1330052608

Kernel/messages log output follows:

Sep  8 00:34:42 dbivr1-l3-sjo kernel: oom-killer: gfp_mask=0xd0
Sep  8 00:34:42 dbivr1-l3-sjo kernel: DMA per-cpu:
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 hot: low 2, high 6, batch 1
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 cold: low 0, high 2, batch 1
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 1 hot: low 2, high 6, batch 1
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 1 cold: low 0, high 2, batch 1
Sep  8 00:34:42 dbivr1-l3-sjo kernel: Normal per-cpu:
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 hot: low 32, high 96, batch 16
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 cold: low 0, high 32, batch 16
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 1 hot: low 32, high 96, batch 16
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 1 cold: low 0, high 32, batch 16
Sep  8 00:34:42 dbivr1-l3-sjo kernel: HighMem per-cpu:
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 hot: low 32, high 96, batch 16
Sep  8 00:34:49 dbivr1-l3-sjo kernel: cpu 0 cold: low 0, high 32, batch 16
Sep  8 00:34:49 dbivr1-l3-sjo kernel: cpu 1 hot: low 32, high 96, batch 16
Sep  8 00:34:49 dbivr1-l3-sjo kernel: cpu 1 cold: low 0, high 32, batch 16
Sep  8 00:34:49 dbivr1-l3-sjo kernel:
Sep  8 00:34:49 dbivr1-l3-sjo kernel: Free pages:        4644kB (832kB HighMem)
Sep  8 00:34:49 dbivr1-l3-sjo kernel: Active:627402 inactive:1406658 dirty:2 writeback:0 unstable:0 free:1161 slab:24425 mapped:384142 pagetables:1461
Sep  8 00:34:49 dbivr1-l3-sjo kernel: DMA free:68kB min:68kB low:84kB high:100kB active:8132kB inactive:0kB present:16384kB pages_scanned:18265 all_unreclaimable? yes
Sep  8 00:34:49 dbivr1-l3-sjo kernel: protections[]: 0 0 0
Sep  8 00:34:49 dbivr1-l3-sjo kernel: Normal free:3744kB min:3756kB low:4692kB high:5632kB active:724812kB inactive:0kB present:901120kB pages_scanned:844317 all_unreclaimable? yes
Sep  8 00:34:49 dbivr1-l3-sjo kernel: protections[]: 0 0 0
Sep  8 00:34:50 dbivr1-l3-sjo kernel: HighMem free:832kB min:512kB low:640kB high:768kB active:1776664kB inactive:5626632kB present:7471104kB pages_scanned:0 all_unreclaimable? no
Sep  8 00:34:50 dbivr1-l3-sjo kernel: protections[]: 0 0 0
Sep  8 00:34:50 dbivr1-l3-sjo kernel: DMA: 1*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68kB
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Normal: 0*4kB 0*8kB 4*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3744kB
Sep  8 00:34:50 dbivr1-l3-sjo kernel: HighMem: 4*4kB 38*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 832kB
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 31322 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3225 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3231 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3232 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3233 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3234 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3235 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3236 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3237 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3238 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3239 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3240 (mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3241 (mysqld.Linux.i6).

etc. ......

How to repeat:
Unknown other than we can consistently recreate under high load for 1-2 weeks of usage.
[27 Oct 2005 14:38] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/31565
[27 Oct 2005 15:11] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/31567
[27 Oct 2005 15:16] Jonas Oreland 
Hm...the leak is present in more situations that orginally specified
[9 Nov 2005 8:47] Jonas Oreland 
Pushed into 5.0.16
  (will fix 4.1 later)
[9 Nov 2005 8:47] Jonas Oreland 
Pushed into 5.0.16
  (will fix 4.1 later)
[22 Nov 2005 16:46] frank homann 
i'd the same problem running a cluster of 3 nodes and the above patch solved it...
is version 5.0.16 including it?
i ask because i didn't find it in the changelog... http://dev.mysql.com/doc/refman/5.0/en/news-5-0-16.html
will version 4.1.16 include it too?

thank you!
[14 Dec 2005 6:35] Jonas Oreland 
pushed into 4.1.17
[15 Dec 2005 13:30] Jon Stephens 
Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html

Additional info:

Changed category to Server:Cluster.

Documented in 4.1.17 and 5.0.16 changelogs.

Closed.
[18 Jan 2006 9:57] Bugs System 
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/33360