Bug #38264 Cluster Backup ends with error 2810
Submitted: 21 Jul 2008 18:15 Modified: 22 Oct 2008 11:11
Reporter: Michael Neubert
Status: Open
Category:Server: Cluster Severity:S2 (Serious)
Version:mysql-5.1-telco-6.2 OS:Any
Assigned to: Target Version:
Tags: mysql-5.1.23 ndb-6.2.14, disk space, Backup, cluster
Triage: Triaged: D2 (Serious) / R6 (Needs Assessment) / E6 (Needs Assessment)

[21 Jul 2008 18:15] Michael Neubert 
Description:
Hello,

we are using a cluster setup of 4 nodes (number of replicas = 2). When the whole cluster
is online a cluster backup writes approximately 12 GB data per node to the local storage
disk, where more than 150 GB free disk space is available.

If we stop one node (e.g. of nodegroup 0) and then start a cluster backup, the one single
node in nodegroup 0 should write 24 GB data to the local disk and the 2 nodes of nodegroup
1 each 12 GB.

With mysql-5.1.22 ndb-6.2.7 this was no problem, but now we get:

- ERROR 2810 "No space left on the device" -

...after writing approximately 16 GB of data to the single node in nodegroup 0 and the
Backup is aborted.

After searching the bugs database I found the following bug report:

http://bugs.mysql.com/bug.php?id=28647

Maybe the patch, that was pushed to 5.1.23-ndb-6.2.11 lead to the described problem,
because this new error code was just introduced in this revision.

Best wishes
Michael

How to repeat:
see scenario above

Suggested fix:
no fix available, just try to avoid disconnected nodes when cluster backup starts, that
writes a lot of data
[22 Oct 2008 10:47] li zhou 
What are the physical memory of your machines.
Can you create a single file whose size is bigger than 16GB in the hosts of nodegroup 0.
Can you privide configuration files.
[22 Oct 2008 11:11] Michael Neubert 
The machines have 16 GB of physical memory. Furthermore we are using disk data tables to
increase the available memory.

Our backup space is using ext3 filesystem. Under kernel 2.4 the file size is limited to
16 GB, but we are using kernel 2.6, where 2 TB should be possible. And we already had
files with a size of 20 GB+.