MySQL Bugs: #42133: ndbd is swapping

Bug #42133	ndbd is swapping
Submitted:	15 Jan 2009 14:50	Modified:	20 Feb 2016 10:41
Reporter:	Cyril SCETBON	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	mysql-5.1.32 ndb-6.3.23	OS:	Linux (debian etch)
Assigned to:	MySQL Verification Team	CPU Architecture:	Any
Tags:	cluster, mysql-5.1.32-telco-6.3.23-telco, ndbd, replication, swap

Description:
We're using dual MySQL Cluster architecture with replication.

Datanodes are swapping (1Go) although we set LockPagesInMainMemory=1 in the ndb manager's configuration. We're using ndb-cluster-connection-pool=10 in mysqld. Can it be caused by the number of connections ?

How to repeat:
cat /proc/sys/vm/swappiness
60

ndbd started and we wait

Suggested fix:
none

please check the ndbd out log it will tell you if pages actually get locked, or if there is some permissions issue or similar.

BR,

Toms

I see no error. I do not see messages saying that pages are locked in memory too.

Can you please supply the log?

BR,

Tomas

ndbd output

Attachment: ndb_3_out.log.gz (application/binary, text), 3.23 KiB.

Please supply the ndb cluster log,

Thanks!

Error messages related to "LockPagesInMemory" will go to the global cluster log, not the data nodes local output log. So we need the ndb_1_cluster.log file from the management node to analyze this any further.

I'd also like to see the cluster configuration file, the systems hardware configuration (esp. total RAM) and an overview of the data nodes memory 
usage, e.g. a snapshot of the "top" output after pressing "M" to get the
process list sorted by memory usage instead of CPU time used.

I've sent you the log of the 2 ndb_mgmd we're using, a current top snapshot even if there is no swap at this time, a memory graphic showing when swap happened and our ndb_mgmd configuration file.

When swap happened we have resolved it by restarting the ndbd daemon.

/proc/memory from one datanode

Attachment: meminfo (application/octet-stream, text), 630 bytes.

Workaround, add more physical memory or reduce memory usage in configuration

I've enough memory according to my configuration. I've forced the lock of pages in memory in my configuration file, so why is it swapping ? For your information I'm not consuming all the memory I've given to ndbds. Do you think it's due to other memory allocations (transactions, redo, etc..) ?

ps shows that ndb uses (or reserved) near 1GB of memory:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                        
 5301 root      -3   0 1098m 209m 3108 S  0.0 10.2   0:18.46 ndbd    

However my configuration is below :

[NDBD DEFAULT]
NoOfReplicas=2
DataDir= /WOO/BASE/data
NoOfFragmentLogFiles=12
FragmentLogFileSize=128M
ODirect=1
#LockPagesInMainMemory=1
RedoBuffer=32M
DataMemory=512M
IndexMemory=64M
#MaxNoOfConcurrentOperations=125000
MaxNoOfConcurrentOperations=25000
RealtimeScheduler=1
LockExecuteThreadToCPU=2
LockMaintThreadsToCPU=3
CompressedLCP=1
CompressedBackup=1
Diskcheckpointspeed=10M
Diskcheckpointspeedinrestart=100M
TimeBetweenGlobalCheckpoints=1000
TimeBetweenLocalCheckpoints=20
BackupLogBufferSize=16M
BackupDataBufferSize=16M
BackupMemory=32M
BackupWriteSize=256K
BackupMaxWriteSize=1M

if I uncomment LockPagesInMainMemory the node is swapping as imho it tries to allocate 1GB in memory that it hasn't.

any idea ? Is it a normal behaviour ?

Does the machine this happens on happens to be a multiprocessor box with NUMA architecture?

yes. However 4 months later I haven't information anymore.

sorry, but the response was no.

Hi,
Any news about this issue?

We have upgraded to 5.1.44 ndb-7.1.4 and set /proc/sys/vm/swappiness to zero but the server is still swapping

Regards

Hi,

4 years later :(, do you still see this problem? 
What version you are running now?

Thanks
Bogdan Kecman

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".