MySQL Bugs: #30556: Memory Usage > 100% When LockPagesInMainMemory=1

Bug #30556	Memory Usage > 100% When LockPagesInMainMemory=1
Submitted:	22 Aug 2007 7:00	Modified:	4 Oct 2007 6:42
Reporter:	Bradley Falzon	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	5.0.45	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
When Upgrading our Cluster from 5.0.21 to 5.0.45, the management node would upgrade fine (when replacing binaries) however, the first node would simply continue to allocate itself more and more memory - resulting in over 100% usage and kernel randomly killing processes. The time between starting to crashing is within a minute, and normally idles at a few percent, then grows to 10, 20, 40, 80, 100 within 15 seconds.

The only way we could stop this was removing the directive LockPagesInMainMemory by commenting out, we tried adjusting all MaxNoOf's, however, no luck.

We were able to reproduce on a test cluster.

How to repeat:
Basically, we reproduced via 2 NDBD Nodes, 1 MySQL Node all version 5.0.21 ~80% Data Memory Usage, ~65% Index Memory Usage reported via all dump 1000. Basically, I just kept re-inserting junk data until it was to that approx usage. But honestly, I wouldnt be surprised if you could reproduce this with next to no data / index memory usage at all.

We upgraded the ndb_mgm and ndb_mgmd binaries and restarted ndb_mgmd.

Because our datadir is /data and binaries are /usr/local/mysql, to upgrade we simply:

cd /usr/local
tar xvzf mysql-5.0.45-etc-etc-etc.tar.gz
mv mysql-5.0.45-etc-etc mysql5045
mv mysql mysql5021
mv mysql5045 mysql
ndbd -n --initial

Run '/usr/bin/top' on the node, and in ndb_mgm <ID> start

Watch closely, and you'll see the memory usage simply sky rocket until the kernel begins to kill random processes. From my experience, kernel usually decides to kill sshd first when your offsite or late at night. :)

Suggested fix:
Investigate the changes between 5.0.21 to 5.0.45 (some reported changes in v 5.0.38 I think).

Based on a few rough calculations, your config.ini looks to allocate 7991MB for each node. If you system has 8GB memory, this would only leave ~10MB for your operating system.

If you lower your DataMemory (ifpossible) or other memory consuming parameters to save a few more hundred MB perhaps it will start properly then.

LockPagesInMemory had suffered from a bug; http://bugs.mysql.com/26454 & http://bugs.mysql.com/25686

To me this looks like the newer version is behaving as expected. Let me know if you believe otherwise.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".