Description:
We filled up the disk on a server and saw this (names adjusted but you get the idea):
2014-10-14 06:10:11 23464 [ERROR] /usr/sbin/mysqld: The table 'MyDailyTable_20141021' is full
2014-10-14 06:10:11 23464 [ERROR] /usr/sbin/mysqld: The table 'MyDailyTable_20150110' is full
2014-10-14 06:10:11 23464 [ERROR] /usr/sbin/mysqld: The table 'MyDailyTable_20150110' is full
2014-10-14 06:10:12 23464 [ERROR] /usr/sbin/mysqld: The table 'MyDailyTable_20150110' is full
2014-10-14 06:10:16 23464 [Warning] Disk is full writing './some_db/some_table.MYD' (Errcode: 28 - No space left on device). Waiting for someone to free space...
2014-10-14 06:10:16 23464 [Warning] Retry in 60 secs. Message reprinted in 600 secs
2014-10-14 06:20:16 23464 [Warning] Disk is full writing './some_db/some_table.MYD' (Errcode: 28 - No space left on device). Waiting for someone to free space...
2014-10-14 06:20:16 23464 [Warning] Retry in 60 secs. Message reprinted in 600 secs
2014-10-14 06:30:16 23464 [Warning] Disk is full writing './some_db/some_table.MYD' (Errcode: 28 - No space left on device). Waiting for someone to free space...
2014-10-14 06:30:16 23464 [Warning] Retry in 60 secs. Message reprinted in 600 secs
2014-10-14 06:40:16 23464 [Warning] Disk is full writing './some_db/some_table.MYD' (Errcode: 28 - No space left on device). Waiting for someone to free space...
2014-10-14 06:40:16 23464 [Warning] Retry in 60 secs. Message reprinted in 600 secs
2014-10-14 06:50:16 23464 [Warning] Disk is full writing './some_db/some_table.MYD' (Errcode: 28 - No space left on device). Waiting for someone to free space...
2014-10-14 06:50:16 23464 [Warning] Retry in 60 secs. Message reprinted in 600 secs
2014-10-14 07:00:16 23464 [Warning] Disk is full writing './some_db/some_table.MYD' (Errcode: 28 - No space left on device). Waiting for someone to free space...
2014-10-14 07:00:16 23464 [Warning] Retry in 60 secs. Message reprinted in 600 secs
2014-10-14 07:10:16 23464 [Warning] Disk is full writing './some_db/some_table.MYD' (Errcode: 28 - No space left on device). Waiting for someone to free space...
2014-10-14 07:10:16 23464 [Warning] Retry in 60 secs. Message reprinted in 600 secs
2014-10-14 07:20:16 23464 [Warning] Disk is full writing './some_db/some_table.MYD' (Errcode: 28 - No space left on device). Waiting for someone to free space...
2014-10-14 07:20:16 23464 [Warning] Retry in 60 secs. Message reprinted in 600 secs
2014-10-14 07:30:16 23464 [Warning] Disk is full writing './some_db/some_table.MYD' (Errcode: 28 - No space left on device). Waiting for someone to free space...
2014-10-14 07:30:16 23464 [Warning] Retry in 60 secs. Message reprinted in 600 secs
I free up some space to resolve the problem but there's no indication that this "issue" was fixed.
It would be most convenient to be able to see that the reported problem has been resolved and at what time that happened.
How to repeat:
Fill up the disk, look in the log file and then add some more space and see that there's no recovery message
Suggested fix:
Add a log entry indicating that the space issue has been resolved so it's clear that the problem has gone away. This may seem like a pointless excercise but if we have a problem and want to resolve it as soon as possible it's not clear that the problem has been fixed and MySQL is working "normally" again unless you look at other metrics like disk i/o, or that the entries in show processlist are no longer stuck/blocked.
