Description:
Both our master and slave servers threw OS error number 5s and crashed. Both systems went down and we had to restart from a snapshot.
Here is the error from the log:
2020-11-19T11:52:05.284803Z 0 [ERROR] InnoDB: Write to file ./ccis/*****.ibdfailed at offset 1977335808, 16384 bytes should have been written, only 0 were written. Operating system error number 5. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2020-11-19T11:52:05.284842Z 0 [ERROR] InnoDB: Error number 5 means 'Input/output error'
2020-11-19T11:52:05.284851Z 0 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html
2020-11-19 11:52:05 0x7fe809587780 InnoDB: Assertion failure in thread 140634565932928 in file fil0fil.cc line 5826
InnoDB: Failing assertion: req_type.is_dblwr_recover() || err == DB_SUCCESS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
11:52:05 UTC - mysqld got signal 6;
A similar error happened in both the master and slave server. We then rolled back to a snapshot and brought things back up and things seemed to work OK.
The actual servers did not throw any errors.
How to repeat:
Not sure, this was a one time occurrence. We are implementing more monitoring to see if there is something we can catch if it happens a next time
