Bug #101940 OS error caused database crash
Submitted: 9 Dec 2020 17:07 Modified: 9 Dec 2020 18:19
Reporter: Brent Creager Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Errors Severity:S3 (Non-critical)
Version:5.7 OS:CentOS
Assigned to: CPU Architecture:Any

[9 Dec 2020 17:07] Brent Creager
Description:
Both our master and slave servers threw OS error number 5s and crashed. Both systems went down and we had to restart from a snapshot.

Here is the error from the log:
2020-11-19T11:52:05.284803Z 0 [ERROR] InnoDB: Write to file ./ccis/*****.ibdfailed at offset 1977335808, 16384 bytes should have been written, only 0 were written. Operating system error number 5. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2020-11-19T11:52:05.284842Z 0 [ERROR] InnoDB: Error number 5 means 'Input/output error'
2020-11-19T11:52:05.284851Z 0 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html
2020-11-19 11:52:05 0x7fe809587780  InnoDB: Assertion failure in thread 140634565932928 in file fil0fil.cc line 5826
InnoDB: Failing assertion: req_type.is_dblwr_recover() || err == DB_SUCCESS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
11:52:05 UTC - mysqld got signal 6;

A similar error happened in both the master and slave server. We then rolled back to a snapshot and brought things back up and things seemed to work OK.

The actual servers did not throw any errors.

How to repeat:
Not sure, this was a one time occurrence. We are implementing more monitoring to see if there is something we can catch if it happens a next time
[9 Dec 2020 18:19] MySQL Verification Team
Thank you for the bug report. To check this bug report we need a repeatable test case. If you will able to provide it please add to this bug report, also if you are not using the latest released version try it too.