Bug #10866 full log partition causes indefinite hang, stops system boot up
Submitted: 25 May 2005 20:29 Modified: 13 May 2010 16:03
Reporter: sean finney Email Updates:
Status: Verified Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S4 (Feature request)
Version:4.1 OS:Linux (debian gnu/linux)
Assigned to: CPU Architecture:Any

[25 May 2005 20:29] sean finney
Description:
(cut and pasted from the debian bug report #310579)

This may apply to other versions of mysql-server as well, but it's
something worth noting either way.  Earlier today while running massive
amounts of inserts and other such things, I noticed a large slow down.

Eventually I tried to restart the mysql server to have it take a few
minutes to get through the shutdown phase, and an indefinite amount of
time to start back up before I killed it.  There were still processes
running when I tried to do a shutdown after as well (which didn't run to
completion).

Eventually I tried to reboot the system, and after waiting for quite a
few minutes, ended up doing a hard reboot.

Then when the system started back up, it hung indefinitely (hours+) at
the init spam where it was loading mysqld.  At this time I oculd not ssh
or do anything but ping the machine.  Ctrl-C at the console did not help
either, I had to boot into single user mode.

Come to find out, my log partition (/var/log is its own partition)
happened to have become full.  And mysqld was trying to write to its log
files, and then going into a "I can't write to my logs, so waiting 30
seconds and trying again" loop.  Which it never gives up on.  Ever.

While letting my log partition get full is certainly a bad idea, the
fact that the mysql server manages to prevent the system from booting at
all should probably be addressed.

How to repeat:
have /var/log (or wherever you have mysql logs going) filled up, and attempt to restart mysql

Suggested fix:
give up on writing to the log files and exit perhaps?  check for ENOSPC?
[1 Jun 2005 15:58] Matthew Lord
We should add a configurable switch to either disable this behavior and/or add a variable to allow the setting of how many times to try the write before giving up and forcing an ASSERT or shutting down.