Description:
A common cause of downtime is a full filesystem.
For MySQL this can be prevented by:
- Setting expire_logs_days (Old binlogs wil be removed)
- Setting innodb_file_per_table (OPTIMIZE TABLE will return space)
- Changing the logrote script /etc/logrotate.d/mysql support-files/mysql-log-rotate so that it checks the correct files.
MySQL Enterprise Monitor does have enough information to check the first 2 causes of excessive disk use, but not for the last one.
If logrotate is configured to rotate /var/log/mysqld.log but the real error log is in /var/lib/mysql/hostname-err.err that that could fill-up the disk. The error log is not really that big, but the slow query log and general log could get big enough to cause issues.
How to repeat:
See Description
Suggested fix:
Add data collection item and rules for logrotate.
Lua script which could be used to create a data collectionn item:
------------------------------
logrotatemysql = "/etc/logrotate.d/mysql"
local fh = io.open(logrotatemysql, "r")
if not fh then
str = string.format("Could not open %s for reading",logrotatemysql)
print(str)
os.exit();
end
for line in fh:lines() do
if string.find(line,"^/") then
s,e = string.find(line,"/[a-z/.-_]*")
logfile = string.sub(line, s, e-1)
fhlog = io.open(logfile)
if not fhlog then
str = string.format("%s does not exist", logfile)
else
str = string.format("%s does exist", logfile)
end
io.close(fhlog)
print(str)
end
end
io.close(fh)
------------------------------
Description: A common cause of downtime is a full filesystem. For MySQL this can be prevented by: - Setting expire_logs_days (Old binlogs wil be removed) - Setting innodb_file_per_table (OPTIMIZE TABLE will return space) - Changing the logrote script /etc/logrotate.d/mysql support-files/mysql-log-rotate so that it checks the correct files. MySQL Enterprise Monitor does have enough information to check the first 2 causes of excessive disk use, but not for the last one. If logrotate is configured to rotate /var/log/mysqld.log but the real error log is in /var/lib/mysql/hostname-err.err that that could fill-up the disk. The error log is not really that big, but the slow query log and general log could get big enough to cause issues. How to repeat: See Description Suggested fix: Add data collection item and rules for logrotate. Lua script which could be used to create a data collectionn item: ------------------------------ logrotatemysql = "/etc/logrotate.d/mysql" local fh = io.open(logrotatemysql, "r") if not fh then str = string.format("Could not open %s for reading",logrotatemysql) print(str) os.exit(); end for line in fh:lines() do if string.find(line,"^/") then s,e = string.find(line,"/[a-z/.-_]*") logfile = string.sub(line, s, e-1) fhlog = io.open(logfile) if not fhlog then str = string.format("%s does not exist", logfile) else str = string.format("%s does exist", logfile) end io.close(fhlog) print(str) end end io.close(fh) ------------------------------