Description:
Two node MySQL cluster setup (version 6.3.17), linux.
We have a mysqldump cronjob that runs a full mysqldump on a database that contains about 130 tables. The whole system has been running for about 5 months, the mysqldump job for about a month. Tonight though all NDB tables in our database disappeared at 03.00 AM (the time the dump runs). Nothing at all in the logs, neither in NDB logs nor MySQL API logs.
API node 1:
2009-04-29 16:27:29 [NdbApi] INFO -- Flushing incomplete GCI:s < 5416531/11
2009-04-29 16:27:29 [NdbApi] INFO -- Flushing incomplete GCI:s < 5416531/11
2009-05-07 08:49:12 [NdbApi] INFO -- Management server closed connection early. It is probably being shut down (or has problems). We will
retry the connection. 1006 Illegal reply from server line: 2659
NDB node 1:
2009-05-06 23:13:28 [ndbd] INFO -- Watchdog: User time: 3395936 System time: 2104647
2009-05-06 23:13:28 [ndbd] WARNING -- Watchdog: Warning overslept 212 ms, expected 100 ms.
2009-05-07 04:20:54 [ndbd] INFO -- Watchdog: User time: 3420918 System time: 2117430
2009-05-07 04:20:54 [ndbd] WARNING -- Watchdog: Warning overslept 211 ms, expected 100 ms.
2009-05-07 04:55:04 [ndbd] INFO -- Watchdog: User time: 3420921 System time: 2117433
2009-05-07 04:55:04 [ndbd] WARNING -- Watchdog: Warning overslept 228 ms, expected 100 ms.
2009-05-07 06:20:28 [ndbd] INFO -- Watchdog: User time: 3420923 System time: 2117444
Same in logs on other node.
How to repeat:
Suggestion: run mysqldump in endless loop against the indicated version of mysql cluster.