MySQL Bugs: #51014: mysqld continues operation without logging when binlogs cannot be written!

Bug #51014	mysqld continues operation without logging when binlogs cannot be written!
Submitted:	9 Feb 2010 8:26	Modified:	2 Jun 2014 10:31
Reporter:	Shane Bester (Platinum Quality Contributor)	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Replication	Severity:	S2 (Serious)
Version:	5.1, 5.4	OS:	Any
Assigned to:	Assigned Account	CPU Architecture:	Any

Description:
If an error occurs that prevents mysqld writing to the binary logs (disk full, readonly filesystem, etc) then the logs are disabled and operations continue.

This can lead to out of sync slaves and improper backups.

This is a feature request to allow a DBA to specify if the server should be shutdown, or continuously retry, or ignore the error.

091124  8:22:45 [ERROR] Could not use /path/to/binlog for logging (error 30). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.

after here, modifications to the database continue without binary logging!

http://ronaldbradford.com/blog/never-let-your-binlog-directory-fill-up-2009-07-15/

How to repeat:
use a non-debug mysqld binary for this test!
enable binary logging and pointlog-bin to another mount.
start server, do an insert.
umount the mount and mount as readonly.
flush logs. (notice the error message now)
continue inserting data. no error, and no logging is performed.

Thank you for the feature request.

See also: WL#1790.

The feature to allow DBA to specify how to act when binlog is not available will not be handled by bug#46166. set back to verified.

Is this a feature request? This issue is very serious once it happens... Nothing (not limited to SQL thread, but from all clients) is binlogged but updates succeed without errors. If this happens on master, slaves won't receive anything.

How can this possibly be a feature request? If mysqld cannot write the binlog then either mysqld must stop or commit & auto-commit statements must fail.

what masters Mark Callaghan and Yoshinori Matsunobu said.

Related to this feature request: bug 68953

Thank you for your bug report. This issue has been addressed in the documentation. The updated documentation will appear on our website shortly.

Documented as follows in the 5.7.5 changelog:
Replication: The new system variable binlog_impossible_mode controls what happens if the server cannot write to the binary log, for example, due to a file error. The default for binlog_impossible_mode is IGNORE_ERROR, meaning the server logs the error, halts logging, and continues updates to the database. Setting this variable to ABORT_SERVER makes the server halt logging and shut down the server if it can not write to the binary log.

And added content is available in the documentation here:
http://dev.mysql.com/doc/refman/5.7/en/replication-options-binary-log.html#sysvar_binlog_i...

$ bzr log -n0 -r 5936
------------------------------------------------------------
revno: 5936
committer: Sujatha Sivakumar <sujatha.sivakumar@oracle.com>
branch nick: Bug11758766_mysql-5.6
timestamp: Wed 2014-05-14 12:01:43 +0530
message:
  Bug#11758766:MYSQLD CONTINUES OPERATION WITHOUT LOGGING WHEN
  BINLOGS CANNOT BE WRITTEN
  
  Problem:
  ========
  If an error occurs that prevents mysqld writing to the
  binary logs (disk full, readonly file system, etc) then the
  logs are disabled and operations continue. This can lead to
  out of sync slaves and improper backups.
  
  Analysis:
  =========
  When binlogging becomes impossible due to readonly file
  system at present the binlog gets closed along with an error
  message in error log which says binlog is turned off for the
  whole duration of the server to fix the cause shutdown the
  server and fix the problem. The master continues without
  binlogging which causes the slave to go out of sync. At
  present there is no way user can control the behaviour so it
  will be better to let the user provide an option to stop the
  server immediately on the error.
  
  Fix:
  ====
  A new option "binlogging_impossible_mode" has been
  introduced whose values are "IGNORE" or "ABORT". When
  binlogging becomes impossible if user sets the variable to
  "ABORT" server will stop if user sets it to "IGNORE" binlog
  will be turned off and server will continue.

$ bzr log -n0 -r 6045
------------------------------------------------------------
revno: 6045
committer: Sujatha Sivakumar <sujatha.sivakumar@oracle.com>
branch nick: Bug11758766_mysql-5.6
timestamp: Mon 2014-06-30 18:46:22 +0530
message:
  Bug#11758766:MYSQLD CONTINUES OPERATION WITHOUT LOGGING WHEN
  BINLOGS CANNOT BE WRITTEN
  
  Fixing a typo.

$ bzr log -n0 -r 6049
------------------------------------------------------------
revno: 6049
committer: Venkata Sidagam <venkata.sidagam@oracle.com>
branch nick: 5.6
timestamp: Tue 2014-07-01 14:03:12 +0530
message:
  Bug#11758766:MYSQLD CONTINUES OPERATION WITHOUT LOGGING WHEN
        BINLOGS CANNOT BE WRITTEN
  
  Post push fix.
  Reverting back the file permissions to 0664

Posted by developer:
 
Reopened by merge, no change required for docs. Reclosing.

The fix for this is a variable? Who was arguing on the side of letting the master continue without logging? What is the default?

See http://dev.mysql.com/doc/refman/5.6/en/replication-options-binary-log.html#sysvar_binlog_e... for the details on default value etc.

As a small update to this issue, the default for binlog_error_action changed to ABORT_SERVER for MySQL 5.7.  Thank you for helping to improve MySQL!

http://dev.mysql.com/doc/refman/5.7/en/replication-options-binary-log.html#sysvar_binlog_e...