MySQL Bugs: #13078: mysqld memleak for ndb

Bug #13078	mysqld memleak for ndb
Submitted:	9 Sep 2005 2:55	Modified:	15 Dec 2005 14:30
Reporter:	Howard Hart
Status:	Closed
Category:	Server: Cluster	Severity:	S1 (Critical)
Version:	4.1,5.0	OS:	Linux (Linux 2.6.12)
Assigned to:	Jonas Oreland	Target Version:

Description:
mysqld and/or ndbd appears to use up all available memory and causes Linux to kill all
mysqld processes. Under high load, mysqld/ndbd only seems to last 1 and 1/2 weeks.
Initial startup shows 7 GB out of 8 GB total RAM available. Mysql logs show no activity
until startup and recovery and no core dumps since Linux is killing the process:

Thu Sep  8 00:36:08 GMT 2005
050908  0:36:09  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
050908  0:36:10  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 443 1324810155.
InnoDB: Doing recovery: scanned up to log sequence number 443 1330052608

Kernel/messages log output follows:

Sep  8 00:34:42 dbivr1-l3-sjo kernel: oom-killer: gfp_mask=0xd0
Sep  8 00:34:42 dbivr1-l3-sjo kernel: DMA per-cpu:
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 hot: low 2, high 6, batch 1
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 cold: low 0, high 2, batch 1
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 1 hot: low 2, high 6, batch 1
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 1 cold: low 0, high 2, batch 1
Sep  8 00:34:42 dbivr1-l3-sjo kernel: Normal per-cpu:
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 hot: low 32, high 96, batch 16
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 cold: low 0, high 32, batch 16
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 1 hot: low 32, high 96, batch 16
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 1 cold: low 0, high 32, batch 16
Sep  8 00:34:42 dbivr1-l3-sjo kernel: HighMem per-cpu:
Sep  8 00:34:42 dbivr1-l3-sjo kernel: cpu 0 hot: low 32, high 96, batch 16
Sep  8 00:34:49 dbivr1-l3-sjo kernel: cpu 0 cold: low 0, high 32, batch 16
Sep  8 00:34:49 dbivr1-l3-sjo kernel: cpu 1 hot: low 32, high 96, batch 16
Sep  8 00:34:49 dbivr1-l3-sjo kernel: cpu 1 cold: low 0, high 32, batch 16
Sep  8 00:34:49 dbivr1-l3-sjo kernel:
Sep  8 00:34:49 dbivr1-l3-sjo kernel: Free pages:        4644kB (832kB HighMem)
Sep  8 00:34:49 dbivr1-l3-sjo kernel: Active:627402 inactive:1406658 dirty:2 writeback:0
unstable:0 free:1161 slab:24425 mapped:384142 pagetables:1461
Sep  8 00:34:49 dbivr1-l3-sjo kernel: DMA free:68kB min:68kB low:84kB high:100kB
active:8132kB inactive:0kB present:16384kB pages_scanned:18265 all_unreclaimable? yes
Sep  8 00:34:49 dbivr1-l3-sjo kernel: protections[]: 0 0 0
Sep  8 00:34:49 dbivr1-l3-sjo kernel: Normal free:3744kB min:3756kB low:4692kB
high:5632kB active:724812kB inactive:0kB present:901120kB pages_scanned:844317
all_unreclaimable? yes
Sep  8 00:34:49 dbivr1-l3-sjo kernel: protections[]: 0 0 0
Sep  8 00:34:50 dbivr1-l3-sjo kernel: HighMem free:832kB min:512kB low:640kB high:768kB
active:1776664kB inactive:5626632kB present:7471104kB pages_scanned:0 all_unreclaimable?
no
Sep  8 00:34:50 dbivr1-l3-sjo kernel: protections[]: 0 0 0
Sep  8 00:34:50 dbivr1-l3-sjo kernel: DMA: 1*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68kB
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Normal: 0*4kB 0*8kB 4*16kB 1*32kB 1*64kB 0*128kB
0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3744kB
Sep  8 00:34:50 dbivr1-l3-sjo kernel: HighMem: 4*4kB 38*8kB 0*16kB 0*32kB 0*64kB 0*128kB
0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 832kB
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 31322
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3225
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3231
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3232
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3233
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3234
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3235
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3236
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3237
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3238
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3239
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3240
(mysqld.Linux.i6).
Sep  8 00:34:50 dbivr1-l3-sjo kernel: Out of Memory: Killed process 3241
(mysqld.Linux.i6).

etc. ......

How to repeat:
Unknown other than we can consistently recreate under high load for 1-2 weeks of usage.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/31565

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/31567

Hm...the leak is present in more situations that orginally specified

Pushed into 5.0.16
  (will fix 4.1 later)

Pushed into 5.0.16
  (will fix 4.1 later)

i'd the same problem running a cluster of 3 nodes and the above patch solved it...
is version 5.0.16 including it?
i ask because i didn't find it in the changelog...
http://dev.mysql.com/doc/refman/5.0/en/news-5-0-16.html
will version 4.1.16 include it too?

thank you!

pushed into 4.1.17

Thank you for your bug report. This issue has been committed to our
source repository of that product and will be incorporated into the
next release.

If necessary, you can access the source repository and build the latest
available version, including the bugfix, yourself. More information 
about accessing the source trees is available at
    http://www.mysql.com/doc/en/Installing_source_tree.html

Additional info:

Changed category to Server:Cluster.

Documented in 4.1.17 and 5.0.16 changelogs.

Closed.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/internals/33360