Bug #37430 LockPagesInMainMemory only locks pages after they've been accessed at least once
Submitted: 16 Jun 2008 16:08 Modified: 22 Jan 2010 18:48
Reporter: Hartmut Holzgraefe Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Cluster: Cluster (NDB) storage engine Severity:S3 (Non-critical)
Version:mysql-5.0 OS:Linux
Assigned to: Jonas Oreland
Tags: >= 5.0.36
Triage: Triaged: D3 (Medium) / R4 (High) / E3 (Medium)

[16 Jun 2008 16:08] Hartmut Holzgraefe
Description:
Even with LockPagesInMainMemory=2 the memory allocated for DataMemory, IndexMemory etc. is not swapped in and locked into RAM right away,
memory pages are only created, swapped in and locked after they have actually been touched at least once.

So after startup even with LockPagesInMainMemory=2 the resident size of a freshly started ndbd process is pretty small compared to its virtual size.

This can lead to massive slowdowns during large insert operations (e.g. ndb_restore) as pages are still only swapped in on demand which may take some time if other memory pages need to be swapped out to make room first (and even if there is enough memory available the allocation and mapping process caused by page fault handling may take some time).

So for the =2 mode all allocated memory should be forced to swap in by e.g. writing a single byte every $pagesize bytes to the allocated buffers.

How to repeat:
Start ndbd processes with different LockPagesInMainMemory settings and rather large Data/IndexMemory values.

Check top or ps output for the ndbd process virtual and resident size.

See that the resident size (RSS) is always way smaller than the virtual size (VSZ) and that it changes only minimally with different lock settings.

Suggested fix:
Touch each allocated Data/IndexMemory page after allocation to make sure it is swapped into main memory and then stays there due to the memlock setting. With LockPagesInMainMemory=2 almost all of the ndbds virtual set should be in the resident set, not only a small fraction.
[12 Nov 2008 14:38] Jonas Oreland
i personally think D4/I4
but...
fixing this for 6.4 involves making sure that system doesnt die from watchdog
when actually swapping all those pages in...testing of this is not trivial
R4/E3
[1 Mar 2009 0:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[18 May 2009 6:47] Oli Sennhauser
I cannot repeat this behaviour (on 6.3.20):

LockPagesInMainMemory 0, 1 and 2:

USER        VSZ    RSS STAT COMMAND
root     408260  94100 Sl   ndbd
root     408260  93812 Sl   ndbd

root     408260 408132 SLl  ndbd
root     408260 408132 SLl  ndbd

root     408260 408240 SLl  ndbd
root     408260 408240 SLl  ndbd
[18 Jun 2009 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[21 Jan 2010 11:19] Jonas Oreland
touch

Attachment: bug37430.patch (application/octet-stream, text), 1.83 KiB.

[21 Jan 2010 15:26] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/97738

3087 Jonas Oreland	2010-01-21
      ndb - bug#37430 - force allocated memory into resident set, by touching a byte on every 4k page
[21 Jan 2010 15:55] Bugs System
Pushed into 5.1.41-ndb-7.1.0 (revid:jonas@mysql.com-20100121155346-uhvjs4swq00hffir) (version source revid:jonas@mysql.com-20100121155346-uhvjs4swq00hffir) (merge vers: 5.1.41-ndb-7.1.0) (pib:16)
[21 Jan 2010 15:56] Bugs System
Pushed into 5.1.41-ndb-7.0.11 (revid:jonas@mysql.com-20100121154838-7ufjzm37rxvz1tml) (version source revid:jonas@mysql.com-20100121154838-7ufjzm37rxvz1tml) (merge vers: 5.1.41-ndb-7.0.11) (pib:16)
[21 Jan 2010 16:00] Jonas Oreland
also pushed to 6.3.31
[22 Jan 2010 18:48] Jon Stephens
Documented bugfix in the NDB 6.3.31 and 7.0.11 changelogs as follows:

        When setting LockPagesInMainMemory, the stated memory was not
        allocated when the node was started, but rather only when the
        memory was used by the data node process for other reasons.

Closed.
[16 Apr 2010 19:51] Oli Sennhauser
I have experienced a different behaviour in memory allocation between 7.0.9 and 7.0.11:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

7.0.7

mysql    10554  1.0  8.1 381584 83840 ?        Sl   21:13   0:00 ndbd
mysql    10635  0.9  8.1 381584 83680 ?        Sl   21:13   0:00 ndbd

7.0.9

mysql    14388  1.9  8.0 379232 82148 ?        Sl   21:15   0:00 ndbd
mysql    14489  2.2  7.9 379232 81948 ?        Rl   21:15   0:00 ndbd

7.0.11

mysql    17304  1.0 30.4 379112 312184 ?       Sl   21:17   0:00 ndbd
mysql    17397  1.2 34.2 378988 351472 ?       Sl   21:17   0:01 ndbd

7.0.13

mysql    15724  3.6 34.3 379476 351920 ?       Sl   11:56   0:03 ndbd
mysql    24948  3.1 30.5 379480 313348 ?       Sl   11:55   0:05 ndbd

ndb_config --q id,lockpagesinmainmemory
10,0 11,0

0: Disables locking. This is the default value.

When setting LockPagesInMainMemory, the stated memory was not allocated when the node was started, but rather only when the memory was used by the data node process for other reasons. (Bug#37430)

I assume with this Bugfix the default behaviour of the value 0 was broken.

Now it is a big mess to work with my Laptop (1 Gbyte of RAM) and MySQL Cluster...

Can you please have a look and let me know what you think?

Would be nice to see the advantages and disadvantages of 1 and 2 in the docu. Shall I file a docu bug for this?

Regards,
Oli