MySQL Bugs: #34508: ndbd: unjustified high iowait

Bug #34508	ndbd: unjustified high iowait
Submitted:	13 Feb 2008 0:26	Modified:	13 Feb 2008 11:46
Reporter:	Christian Kujau	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S5 (Performance)
Version:	5.1.22	OS:	Linux (2.6.18-4-amd64)
Assigned to:		CPU Architecture:	Any

Description:
We have an 4 node cluster (2 replicas) and we noticed high iowait times on 2 nodes (1 whole replica). Even when taking 1 node out of the cluster but with ndbd still running, iowait never drops below 15%, it's constantly writing (not reading) "something" to the disk, with 0.5-1.5MB/s, even when it's got "nothing to do" and should be idle. Network traffic to/from the box at this time is < 20KB/s.

Apart from ndbd a few other daemons (cron, sshd) are running on the box, but iowait persists even when shutting down these daemons and only stops when terminating ndbd. strace(1)'ing ndbd seemed possible, but then strace hung and had to be killed with -9. I played around with gdb(1) but I'm to inexperienced how to get useful(!) information from a running (and 2GB consuming) process.

I've collected a few details from the box as well as the .tar.bz2 from ndb_error_reporter: http://nerdbynature.de/bits/ndbd-5.1.22/

Any hints as to what exactly is causing the writes?

Thanks,
Christian.

How to repeat:
1) start ndbd
2) wait, untils it's started...then ndbd begins to write with ~1MB/s to the disk.

We're sorry, but the bug system is not the appropriate forum for asking help on using MySQL products. Your problem is not the result of a bug.

Support on using our products is available both free in our forums at http://forums.mysql.com/ and for a reasonable fee direct from our skilled support engineers at http://www.mysql.com/support/

What you are experiencing is the local checkpoint mechanism
that dumps the whole cluster memory contents to disk on a
regular basis for recovery purpose. Depending on how you
configured TimeBetweenLocalCheckpoints this may indeed 
happen in a continuous way ...

Thank you for your interest in MySQL.