Bug #84191 | redo log corrupted after btrfs filesystem no space left on device | ||
---|---|---|---|
Submitted: | 13 Dec 2016 18:26 | Modified: | 24 Feb 2017 11:06 |
Reporter: | Philipp Gassmann | Email Updates: | |
Status: | Not a Bug | Impact on me: | |
Category: | MySQL Server: InnoDB storage engine | Severity: | S4 (Feature request) |
Version: | 5.7.16 | OS: | Ubuntu (14.04) |
Assigned to: | MySQL Verification Team | CPU Architecture: | Any |
Tags: | corruption, filesystem full, no space |
[13 Dec 2016 18:26]
Philipp Gassmann
[9 Feb 2017 16:08]
MySQL Verification Team
Hi! Since its inception in MySQL 4.0, InnoDB and other storage engines do not and did not check for the free space on the disk, for any of its components. Simply, hard storage is very cheap these days and cases like this are extremely rare. Still, I consider this a valid feature request. Also, I will write to the Documentation team that this behavior gets documented. Verified as a feature request.
[24 Feb 2017 11:06]
MySQL Verification Team
Hi Philipp, I'm sorry but I can't reproduce this on 2 different machines, with btrfs (mounted with "default options" - rw, realatime, seclabel, space_cache, subvolid=5, subvol=/) using kernel 4.7.7 and 3.10.0 ... the only thing I was able to corrupt is myisam table! Pushing data into innodb did not corrupt anything. I tried inserting data in one, 2 and 4 threads in parallel. The mysql crashes (as expected) when disk is full but you remove enough free space and mysql starts right up without any issues. Nothing corrupted. Made over 20 crashes today with different scenario and never had a corruption. Dunno if this had to do with btrfs bug or what else, but I was not able to reproduce it on two different machines with all scenarios I could think of. The redo space is preallocated and it's a "real file" not a sparse one so the only bug here can be the btrfs bug, there's nothing we can do about it. take care Bogdan Kecman
[15 Mar 2019 19:45]
Leho Kraav
This is certainly a real thing, as I just encountered exact same scenario, all the way to details, but on MariaDB 10.2, kernel 4.14.14. Only `innodb_force_recovery = 6` helped. ``` 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Uses event mutexes 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Compressed tables use zlib 1.2.11 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Using Linux native AIO 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Number of pools: 1 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Using SSE2 crc32 instructions 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Completed initialization of buffer pool 2019-03-15 21:41:42 140309522929408 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Highest supported file format is Barracuda. 2019-03-15 21:41:42 140313413187648 [ERROR] InnoDB: Invalid redo log header checksum. 2019-03-15 21:41:42 140313413187648 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption 2019-03-15 21:41:42 140313413187648 [Note] InnoDB: Starting shutdown... 2019-03-15 21:41:42 140313413187648 [ERROR] Plugin 'InnoDB' init function returned error. 2019-03-15 21:41:42 140313413187648 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2019-03-15 21:41:42 140313413187648 [Note] Plugin 'FEEDBACK' is disabled. 2019-03-15 21:41:42 140313413187648 [ERROR] Unknown/unsupported storage engine: InnoDB 2019-03-15 21:41:42 140313413187648 [ERROR] Aborting 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Started in read only mode 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Uses event mutexes 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Compressed tables use zlib 1.2.11 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Using Linux native AIO 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Number of pools: 1 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Using SSE2 crc32 instructions 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Disabling background log and ibuf IO write threads. 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Completed initialization of buffer pool 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: Highest supported file format is Barracuda. 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: innodb_force_recovery=6 skips redo log apply 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: 5.7.25 started; log sequence number 0 2019-03-15 21:41:52 139657289590848 [Note] InnoDB: !!! innodb_force_recovery is set to 6 !!! 2019-03-15 21:41:52 139657289590848 [Note] Plugin 'FEEDBACK' is disabled. 2019-03-15 21:41:52 139657289590848 [Note] Recovering after a crash using gusto-bin 2019-03-15 21:41:52 139657289590848 [Note] Starting crash recovery... 2019-03-15 21:41:52 139657289590848 [Note] Crash recovery finished. ```
[16 Mar 2019 0:20]
MySQL Verification Team
Hi Leho Kraav, While I can reproduce this on maria I was not able to reproduce this on mysql 5.7.24/5.7.25 and this is mysql bugs database so being able to reproduce with maria is 100% irrelevant info. Also, as Sinisa already pointed out - MySQL since day 1 never cared about free disk space, it's system admin's / dba's work, you need to setup monitoring, use MySQL Enterprise Monitor or whatever else.. but you should not allow your system to get to position your datadir partition is full. all best Bogdan
[16 Mar 2019 14:19]
Leho Kraav
Thanks for the confirmation, Bogdan. "Irrelevant" from one aspect, perhaps, but probably still useful from a global know-how perspective. Large majority of people probably are not aware of how far codebases have diverged by now, and sometimes there's only a very limited no of highly similar cases available on search engines. Y, I will be attending to disk space issues for sure, btrfs snapshotting implementation clearly had weak spots.
[16 Mar 2019 19:06]
MySQL Verification Team
One should really monitor one's RDBMS for free space. That is one of the first, most basic, monitors one sysadmin and one dba need to set up. For me, personally, a "something failed 'cause partition was full" is a huge fail of whoever is in charge of that system, irrelevant if it ended up failing your RDBMS or any other service on affected system. all best Bogdan