| Bug #42133 | ndbd is swapping | ||
|---|---|---|---|
| Submitted: | 15 Jan 15:50 | Modified: | 10 Jul 16:48 |
| Reporter: | Cyril SCETBON | ||
| Status: | Open | ||
| Category: | Server: Cluster | Severity: | S2 (Serious) |
| Version: | mysql-5.1.32 ndb-6.3.23 | OS: | Linux (debian etch) |
| Assigned to: | Hartmut Holzgraefe | Target Version: | |
| Tags: | mysql-5.1.32-telco-6.3.23-telco, replication, cluster, swap, ndbd | ||
| Triage: | Triaged: D2 (Serious) / R6 (Needs Assessment) / E6 (Needs Assessment) | ||
[15 Jan 15:50]
Cyril SCETBON
[15 Jan 15:54]
Tomas Ulin
please check the ndbd out log it will tell you if pages actually get locked, or if there is some permissions issue or similar. BR, Toms
[15 Jan 16:08]
Cyril SCETBON
I see no error. I do not see messages saying that pages are locked in memory too.
[15 Jan 16:13]
Tomas Ulin
Can you please supply the log? BR, Tomas
[15 Jan 16:13]
Cyril SCETBON
ndbd output
Attachment: ndb_3_out.log.gz (application/binary, text), 3.23 KiB.
[22 Jan 15:22]
Jonathan Miller
Please supply the ndb cluster log, Thanks!
[22 Jan 19:47]
Hartmut Holzgraefe
Error messages related to "LockPagesInMemory" will go to the global cluster log, not the data nodes local output log. So we need the ndb_1_cluster.log file from the management node to analyze this any further. I'd also like to see the cluster configuration file, the systems hardware configuration (esp. total RAM) and an overview of the data nodes memory usage, e.g. a snapshot of the "top" output after pressing "M" to get the process list sorted by memory usage instead of CPU time used.
[22 Jan 21:11]
Cyril SCETBON
I've sent you the log of the 2 ndb_mgmd we're using, a current top snapshot even if there is no swap at this time, a memory graphic showing when swap happened and our ndb_mgmd configuration file. When swap happened we have resolved it by restarting the ndbd daemon.
[22 Jan 21:12]
Cyril SCETBON
/proc/memory from one datanode
Attachment: meminfo (application/octet-stream, text), 630 bytes.
[25 Mar 12:03]
Jonathan Miller
Workaround, add more physical memory or reduce memory usage in configuration
[25 Mar 14:58]
Cyril SCETBON
I've enough memory according to my configuration. I've forced the lock of pages in memory in my configuration file, so why is it swapping ? For your information I'm not consuming all the memory I've given to ndbds. Do you think it's due to other memory allocations (transactions, redo, etc..) ?
[10 Jul 16:48]
Cyril SCETBON
ps shows that ndb uses (or reserved) near 1GB of memory:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5301 root -3 0 1098m 209m 3108 S 0.0 10.2 0:18.46 ndbd
However my configuration is below :
[NDBD DEFAULT]
NoOfReplicas=2
DataDir= /WOO/BASE/data
NoOfFragmentLogFiles=12
FragmentLogFileSize=128M
ODirect=1
#LockPagesInMainMemory=1
RedoBuffer=32M
DataMemory=512M
IndexMemory=64M
#MaxNoOfConcurrentOperations=125000
MaxNoOfConcurrentOperations=25000
RealtimeScheduler=1
LockExecuteThreadToCPU=2
LockMaintThreadsToCPU=3
CompressedLCP=1
CompressedBackup=1
Diskcheckpointspeed=10M
Diskcheckpointspeedinrestart=100M
TimeBetweenGlobalCheckpoints=1000
TimeBetweenLocalCheckpoints=20
BackupLogBufferSize=16M
BackupDataBufferSize=16M
BackupMemory=32M
BackupWriteSize=256K
BackupMaxWriteSize=1M
if I uncomment LockPagesInMainMemory the node is swapping as imho it tries to allocate
1GB in memory that it hasn't.
any idea ? Is it a normal behaviour ?
