Bug #39711 Running falcon_bug_34351_C shows increasing memory usage
Submitted: 28 Sep 2008 15:54 Modified: 9 Jan 2009 13:25
Reporter: Hakan Küçükyılmaz Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Falcon storage engine Severity:S2 (Serious)
Version:6.0-falcon-team, 6.0.8-bzr OS:Any
Assigned to: Christopher Powers CPU Architecture:Any
Tags: F_INDEX
Triage: Triaged: D1 (Critical)

[28 Sep 2008 15:54] Hakan Küçükyılmaz
Description:
Running falcon_bug_34351_C shows increasing memory usage.

How to repeat:
Run the test several times in a row like

./mysql-test-run.pl --force --mem --skip-ndb --suite=falcon falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C

and watch steady memory usage increase.
[28 Sep 2008 16:31] Hakan Küçükyılmaz
Memory usage details after eleven runs:

hakan@lu0011:~/work/mysql/mysql-6.0-falcon-team-local-master$ while (true); do echo "select file, line, space_in_use from information_schema.falcon_system_memory_detail order by space_in_use desc limit 10" | mysql -uroot -P9306 -h127.0.0.1 test ; sleep 10; done 

file                    line    space_in_use
Index.cpp               277     613989376
SerialLog.cpp           869     37475328
TransactionManager.cpp  136     24729792
SerialLog.cpp           190     20972064
Cache.cpp               152     4198432
Transaction.cpp         169     2482816
TransactionManager.cpp  129     2453888
Cache.cpp               939     2048064
DeferredIndex.cpp        98      524928
Cache.cpp               140     417832

file                    line    space_in_use
Index.cpp               277     652807168
TransactionManager.cpp  136     73862272
SerialLog.cpp           869     46433280
SerialLog.cpp           190     20972064
Transaction.cpp         169     7454112
TransactionManager.cpp  129     7325184
Cache.cpp               152     4198432
Cache.cpp               939     2048064
DeferredIndex.cpp        98      524928
Cache.cpp               140     417832

file                    line    space_in_use
Index.cpp               277     682689280
TransactionManager.cpp  136     111413568
SerialLog.cpp           869     33787776
SerialLog.cpp           190     20972064
Transaction.cpp         169     11264896
TransactionManager.cpp  129     11059520
Cache.cpp               152     4198432
Cache.cpp               939     2048064
DeferredIndex.cpp       98      524928
Cache.cpp               140     417832
[28 Sep 2008 16:33] Valeriy Kravchuk
Verified with 6.0.8 from bzr also:

openxs@suse:/home2/openxs/dbs/6.0/mysql-test> ./mysql-test-run --force --mem --skip-ndb --suite=falcon falcon_bug_34351_C  falcon_bug_34351_C  falcon_bug_34351_C  falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C
Logging: ./mysql-test-run --force --mem --skip-ndb --suite=falcon falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C falcon_bug_34351_C
MySQL Version 6.0.8
Using dynamic switching of binlog format
Using tmpfs in /dev/shm
Skipping ndbcluster
Setting mysqld to support SSL connections
Binaries are debug compiled
Using MTR_BUILD_THREAD      = 0
Using MASTER_MYPORT         = 9306
Using MASTER_MYPORT1        = 9307
Using SLAVE_MYPORT          = 9308
Using SLAVE_MYPORT1         = 9309
Using SLAVE_MYPORT2         = 9310
Killing Possible Leftover Processes
Removing Stale Files
Creating Directories
Symlinking 'var' to '/dev/shm/var'
Installing Master Database
Saving snapshot of installed databases
=======================================================

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

falcon.falcon_bug_34351_C      [ pass ]          52905
falcon.falcon_bug_34351_C      [ pass ]          58634
falcon.falcon_bug_34351_C      [ pass ]          72359
falcon.falcon_bug_34351_C      [ pass ]          78257
falcon.falcon_bug_34351_C      [ pass ]          74488
falcon.falcon_bug_34351_C      [ pass ]          79137
falcon.falcon_bug_34351_C      [ pass ]          89919
mysql-test-run: WARNING: got INT signal, cleaning up.....
Stopping All Servers
mysql-test-run: *** ERROR: We die from ^C signal from user

At the same time free shows:

openxs@suse:/home2/openxs/dbs/5.1> free -s 30 | tee free.log
             total       used       free     shared    buffers     cached
Mem:        256300     252864       3436          0        472      26952
-/+ buffers/cache:     225440      30860
Swap:       514040     278832     235208

             total       used       free     shared    buffers     cached
Mem:        256300     252620       3680          0        540      20840
-/+ buffers/cache:     231240      25060
Swap:       514040     314544     199496

             total       used       free     shared    buffers     cached
Mem:        256300     252808       3492          0        592      23808
-/+ buffers/cache:     228408      27892
Swap:       514040     322904     191136

             total       used       free     shared    buffers     cached
Mem:        256300     252592       3708          0        420      28148
-/+ buffers/cache:     224024      32276
Swap:       514040     351212     162828

             total       used       free     shared    buffers     cached
Mem:        256300     252740       3560          0        400      22136
-/+ buffers/cache:     230204      26096
Swap:       514040     349460     164580

             total       used       free     shared    buffers     cached
Mem:        256300     252764       3536          0        592      22492
-/+ buffers/cache:     229680      26620
Swap:       514040     390500     123540

             total       used       free     shared    buffers     cached
Mem:        256300     252028       4272          0        496      27264
-/+ buffers/cache:     224268      32032
Swap:       514040     399624     114416

             total       used       free     shared    buffers     cached
Mem:        256300     252608       3692          0        332      19984
-/+ buffers/cache:     232292      24008
Swap:       514040     429276      84764

             total       used       free     shared    buffers     cached
Mem:        256300     252280       4020          0        500      22732
-/+ buffers/cache:     229048      27252
Swap:       514040     432972      81068

             total       used       free     shared    buffers     cached
Mem:        256300     252724       3576          0        388      18136
-/+ buffers/cache:     234200      22100
Swap:       514040     455104      58936

             total       used       free     shared    buffers     cached
Mem:        256300     252032       4268          0        356      17856
-/+ buffers/cache:     233820      22480
Swap:       514040     463624      50416

             total       used       free     shared    buffers     cached
Mem:        256300     252096       4204          0        600      19572
-/+ buffers/cache:     231924      24376
Swap:       514040     484244      29796

             total       used       free     shared    buffers     cached
Mem:        256300     253224       3076          0        388      21872
-/+ buffers/cache:     230964      25336
Swap:       514040     490240      23800

             total       used       free     shared    buffers     cached
Mem:        256300     252584       3716          0        336      18404
-/+ buffers/cache:     233844      22456
Swap:       514040     507908       6132

             total       used       free     shared    buffers     cached
Mem:        256300     252380       3920          0        496      23288
-/+ buffers/cache:     228596      27704
Swap:       514040     507552       6488

             total       used       free     shared    buffers     cached
Mem:        256300     253160       3140          0        476      26968
-/+ buffers/cache:     225716      30584
Swap:       514040     514040          0

Then I was forced to stop testing...
[1 Oct 2008 15:43] Kevin Lewis
Commenting out this line in Transaction::add(
-	deferredIndex->addRef();
+//	deferredIndex->addRef();

will eliminate this memory leak.  But this is a real reference to the DeferredIndex, so the addRef should remain and the true location of the missing call to DeferredIndex::releaseRef should be found.
[2 Oct 2008 0:12] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/54992

2845 Christopher Powers	2008-10-01
      Bug#39711 "Running falcon_bug_34351_C shows increasing memory usage"
      Bug#39795 "Falcon: Online add index does not support index with non-null columns"
      Bug#39796 "Falcon: Reference count decrement not atomic"
[2 Oct 2008 0:13] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/54993

2845 Christopher Powers	2008-10-01
      Bug#39711 "Running falcon_bug_34351_C shows increasing memory usage"
      Bug#39795 "Falcon: Online add index does not support index with non-null columns"
      Bug#39796 "Falcon: Reference count decrement not atomic"
[2 Oct 2008 0:16] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/54994

2845 Christopher Powers	2008-10-01
      Bug#39711 "Running falcon_bug_34351_C shows increasing memory usage"
      Bug#39795 "Falcon: Online add index does not support index with non-null columns"
      Bug#39796 "Falcon: Reference count decrement not atomic"
[2 Oct 2008 23:54] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/55161

2849 Christopher Powers	2008-10-02
      Bug#39711, "Running falcon_bug_34351_C shows increasing memory usage"
[30 Nov 2008 23:17] Hakan Küçükyılmaz
Debug build with latest code from mysql-6.0-falcon bzr tree shows stable memory usage around 325 - 389 MB and run time around 15 seconds.

However, when using a non-debug build the run time decreases to 4 - 6 seconds with spikes at around 14 seconds and memory usage starting at 234 MB and going up to 696 MB.

MySQL Version 6.0.9
Using dynamic switching of binlog format
Using tmpfs in /dev/shm
Skipping ndbcluster
Setting mysqld to support SSL connections
Using MTR_BUILD_THREAD      = 0
Using MASTER_MYPORT         = 9306
Using MASTER_MYPORT1        = 9307
Using SLAVE_MYPORT          = 9308
Using SLAVE_MYPORT1         = 9309
Using SLAVE_MYPORT2         = 9310
Killing Possible Leftover Processes
Removing Stale Files
Creating Directories
Symlinking 'var' to '/dev/shm/var'
Installing Master Database
Saving snapshot of installed databases
=======================================================

TEST                           RESULT         TIME (ms)
-------------------------------------------------------

falcon.falcon_bug_34351_C      [ pass ]           4714
falcon.falcon_bug_34351_C      [ pass ]           4505
falcon.falcon_bug_34351_C      [ pass ]           4662
falcon.falcon_bug_34351_C      [ pass ]           4663
falcon.falcon_bug_34351_C      [ pass ]           4965
falcon.falcon_bug_34351_C      [ pass ]           4999
falcon.falcon_bug_34351_C      [ pass ]           5616
falcon.falcon_bug_34351_C      [ pass ]           5730
falcon.falcon_bug_34351_C      [ pass ]           6253
falcon.falcon_bug_34351_C      [ pass ]           6573
falcon.falcon_bug_34351_C      [ pass ]          14420
falcon.falcon_bug_34351_C      [ pass ]           4306
falcon.falcon_bug_34351_C      [ pass ]           4482
falcon.falcon_bug_34351_C      [ pass ]           4925
falcon.falcon_bug_34351_C      [ pass ]           4689
falcon.falcon_bug_34351_C      [ pass ]           4934
falcon.falcon_bug_34351_C      [ pass ]           5576
falcon.falcon_bug_34351_C      [ pass ]           5798
falcon.falcon_bug_34351_C      [ pass ]           5596
falcon.falcon_bug_34351_C      [ pass ]           5694
falcon.falcon_bug_34351_C      [ pass ]          14241
falcon.falcon_bug_34351_C      [ pass ]           4638
falcon.falcon_bug_34351_C      [ pass ]           4724
falcon.falcon_bug_34351_C      [ pass ]           4574
falcon.falcon_bug_34351_C      [ pass ]           5133
falcon.falcon_bug_34351_C      [ pass ]           5121
falcon.falcon_bug_34351_C      [ pass ]           5515
falcon.falcon_bug_34351_C      [ pass ]           5604
falcon.falcon_bug_34351_C      [ pass ]           6069
falcon.falcon_bug_34351_C      [ pass ]           6284
falcon.falcon_bug_34351_C      [ pass ]          12573
falcon.falcon_bug_34351_C      [ pass ]           4609
falcon.falcon_bug_34351_C      [ pass ]           4726
falcon.falcon_bug_34351_C      [ pass ]           4780
falcon.falcon_bug_34351_C      [ pass ]           4927
falcon.falcon_bug_34351_C      [ pass ]           4949
falcon.falcon_bug_34351_C      [ pass ]           5264
falcon.falcon_bug_34351_C      [ pass ]           5550
...
...
...

falcon.falcon_bug_34351_C      [ pass ]           5507
falcon.falcon_bug_34351_C      [ pass ]           6136
falcon.falcon_bug_34351_C      [ pass ]          14797
falcon.falcon_bug_34351_C      [ pass ]           4507
falcon.falcon_bug_34351_C      [ pass ]           4710
falcon.falcon_bug_34351_C      [ pass ]           4580
falcon.falcon_bug_34351_C      [ pass ]           4785
falcon.falcon_bug_34351_C      [ pass ]           5009
falcon.falcon_bug_34351_C      [ pass ]           5356
falcon.falcon_bug_34351_C      [ pass ]           5440
falcon.falcon_bug_34351_C      [ pass ]           5964
falcon.falcon_bug_34351_C      [ pass ]           5867
falcon.falcon_bug_34351_C      [ pass ]          14159
falcon.falcon_bug_34351_C      [ pass ]           4380
falcon.falcon_bug_34351_C      [ pass ]           4822
falcon.falcon_bug_34351_C      [ pass ]           5083
falcon.falcon_bug_34351_C      [ pass ]           4734
falcon.falcon_bug_34351_C      [ pass ]           5356
falcon.falcon_bug_34351_C      [ pass ]           5772
falcon.falcon_bug_34351_C      [ pass ]           5562
falcon.falcon_bug_34351_C      [ pass ]           6373
falcon.falcon_bug_34351_C      [ pass ]           6562
falcon.falcon_bug_34351_C      [ pass ]          13645
falcon.falcon_bug_34351_C      [ pass ]           4586
falcon.falcon_bug_34351_C      [ pass ]           4722
falcon.falcon_bug_34351_C      [ pass ]           4834
falcon.falcon_bug_34351_C      [ pass ]           5361
falcon.falcon_bug_34351_C      [ pass ]           5571
falcon.falcon_bug_34351_C      [ pass ]           5943
falcon.falcon_bug_34351_C      [ pass ]           6021
falcon.falcon_bug_34351_C      [ pass ]           6094
falcon.falcon_bug_34351_C      [ pass ]          12983
falcon.falcon_bug_34351_C      [ pass ]           4632
falcon.falcon_bug_34351_C      [ pass ]           4527
falcon.falcon_bug_34351_C      [ pass ]           4754
falcon.falcon_bug_34351_C      [ pass ]           4977
falcon.falcon_bug_34351_C      [ pass ]           5133
falcon.falcon_bug_34351_C      [ pass ]           5771
falcon.falcon_bug_34351_C      [ pass ]           6036
falcon.falcon_bug_34351_C      [ pass ]           6133
falcon.falcon_bug_34351_C      [ pass ]           6571
falcon.falcon_bug_34351_C      [ pass ]          14829
falcon.falcon_bug_34351_C      [ pass ]           4745
falcon.falcon_bug_34351_C      [ pass ]           4803
falcon.falcon_bug_34351_C      [ pass ]           5079
falcon.falcon_bug_34351_C      [ pass ]           4924
falcon.falcon_bug_34351_C      [ pass ]           5493
falcon.falcon_bug_34351_C      [ pass ]           5454
falcon.falcon_bug_34351_C      [ pass ]           5314
falcon.falcon_bug_34351_C      [ pass ]           5896
falcon.falcon_bug_34351_C      [ pass ]          14365
falcon.falcon_bug_34351_C      [ pass ]           4570
falcon.falcon_bug_34351_C      [ pass ]           4425
falcon.falcon_bug_34351_C      [ pass ]           4587
falcon.falcon_bug_34351_C      [ pass ]           4960
falcon.falcon_bug_34351_C      [ pass ]           5299
falcon.falcon_bug_34351_C      [ pass ]           5064
falcon.falcon_bug_34351_C      [ pass ]           5419
falcon.falcon_bug_34351_C      [ pass ]           6208
falcon.falcon_bug_34351_C      [ pass ]           5876
falcon.falcon_bug_34351_C      [ pass ]          14037
falcon.falcon_bug_34351_C      [ pass ]           4312
falcon.falcon_bug_34351_C      [ pass ]           4678
falcon.falcon_bug_34351_C      [ pass ]           4777
falcon.falcon_bug_34351_C      [ pass ]           5084
falcon.falcon_bug_34351_C      [ pass ]           5332
falcon.falcon_bug_34351_C      [ pass ]           5925
falcon.falcon_bug_34351_C      [ pass ]           5637
falcon.falcon_bug_34351_C      [ pass ]           6042
falcon.falcon_bug_34351_C      [ pass ]           6180
falcon.falcon_bug_34351_C      [ pass ]          14101
-------------------------------------------------------
Stopping All Servers
All 167 tests were successful.
The servers were restarted 1 times
Spent 1023.631 of 1031 seconds executing testcases
[1 Dec 2008 14:16] John Embretsen
I have verified Hakan's latest observations regarding (relative) running time and memory usage in non-debug vs. debug builds (tested on Linux, using current mysql-6.0-falcon-team bzr branch). Debug build showed stable running times and memory usage, while the non-debug build showed spikes in running time and increasing memory usage.

In my case the difference between the debug and non-debug build was that the debug build was configured with the "--with-debug=full" option.

Hakan, feel free to re-open this bug or file a new one as you see fit.
[9 Jan 2009 13:25] MC Brown
Internal only. No documentation needed.