MySQL Bugs: #31920: MySQL Cluster crashes if there is not enough disk space for backup files

Bug #31920	MySQL Cluster crashes if there is not enough disk space for backup files
Submitted:	29 Oct 2007 16:02	Modified:	17 Mar 2008 16:12
Reporter:	Oli Sennhauser	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Cluster: Cluster (NDB) storage engine	Severity:	S2 (Serious)
Version:	6.0.2 / cge-6.2.6 / 5.1.22	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
MySQL Cluster crashes repeatedly when there is no more free diskspace on the BackupDataDir location (separated backup and cluster files!!!)
This happens at will...

How to repeat:
Set BackupDataDir to other location than the cluster files are
fill the BackupDataDir with dd if=/dev/zero of=... so it is nearly full
do backup

Suggested fix:
backup should abort but not crash the server.

tried to reproduce with 5.1.22 first, this works (or better: fails) 
as expected though, the backup is aborted with an error message
but the ndbd nodes keep on running just fine ...

Hi Harmut

It was clearly reproducible also on 5.1.22:

The free space on /backup came down from around 90 MB to 17... The cluster is located on /home/mysql

Filesystem            Size  Used Avail Use% Mounted on
/dev/hda8             9.8G  9.4G   17M 100% /backup
/dev/hdc6             5.6G  4.8G  526M  91% /home/mysql

# ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> start backup
Connected to Management Server at: localhost:1186
Waiting for completed, this may take several minutes
Node 10: Backup 2 started from node 1
Node 10: Backup 2 started from node 1 completed
 StartGCP: 708 StopGCP: 711
 #Records: 394154 #LogRecords: 0
 Data: 36108864 bytes Log: 0 bytes
ndb_mgm>
ndb_mgm>
ndb_mgm> start backup
Waiting for completed, this may take several minutes
Node 10: Backup 3 started from node 1
Node 10: Backup 3 started from node 1 completed
 StartGCP: 713 StopGCP: 716
 #Records: 394154 #LogRecords: 0
 Data: 36108864 bytes Log: 0 bytes
ndb_mgm>
ndb_mgm>
ndb_mgm> start backup
Waiting for completed, this may take several minutes
Node 10: Backup 4 started from node 1
Node 10: Backup 4 started from 1 has been aborted. Error: 2810
Backup failed
*  3001: Could not start backup
*        Unknown error code: Unknown result: Unknown error code
ndb_mgm> Node 13: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
Node 12: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
Node 10: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.
Node 11: Forced node shutdown completed. Caused by error 2341: 'Internal program error (failed ndbrequire)(Internal error, programming error or missing error message, please report a bug). Temporary error, restart node'.

ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=10 (not connected, accepting connect from master)
id=11 (not connected, accepting connect from master)
id=12 (not connected, accepting connect from master)
id=13 (not connected, accepting connect from master)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @master  (Version: 5.1.22)

[mysqld(API)]   5 node(s)
id=14 (not connected, accepting connect from any host)
id=15 (not connected, accepting connect from any host)
id=16 (not connected, accepting connect from any host)
id=17 (not connected, accepting connect from any host)
id=18 (not connected, accepting connect from any host)

Please, try to repeat with a newer version, 5.1.23-rc, and inform about the results.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

Hello,

the problem mentionned seems to be the same as:

http://bugs.mysql.com/bug.php?id=28647

...that was solved in 5.1.24.

Beste wishes
Michael