MySQL Bugs: #69476: Database page corruption on disk or a failed

Bug #69476	Database page corruption on disk or a failed
Submitted:	14 Jun 2013 17:05	Modified:	25 Apr 2018 9:59
Reporter:	Jan F	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	5.5.29	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
I've been experiencing this problem ever since I've moved to ssd, on both of my servers (each has separate mysql server). innodb_file_per_table is on. Each server has two INTEL SSDSC2CW240A3 in md raid 1 with ext4 filesystem (all parameters are default). At least once a month this problem occurs (and then I fix it with innodb_force_recovery -> mysql dump table > restore).
Also when the problem occurs
a) Server crashes every time when corrupted table is accessed making access to all other tables impossible and corrupting myisam tables. I don't think that should be a way to handle such error with innodb_file_per_table on.
b) mysql log is filled with a log of hex data which fills all drive space in minutes, paralizyng all software that want to write to that drive.
c) I have no idea what is causing corruption and that is the main problem...

How to repeat:
130614 17:37:01 InnoDB: Page checksum 2363010735 (32bit_calc: 3144027128), prior-to-4.0.14-form checksum 4231358755
InnoDB: stored checksum 2495339155, prior-to-4.0.14-form stored checksum 4231358755
InnoDB: Page lsn 158 2649077705, low 4 bytes of lsn at page end 2649077705
InnoDB: Page number (if stored to page already) 27863,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 2490382
InnoDB: Page may be an index page where index id is 3999214
InnoDB: (index "PRIMARY" of table "dbs107_dbajosiebie"."event_views")
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 27863.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 27863.
InnoDB: You may have to recover from a backup.
130614 17:37:01 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 94bbda9300 <lot of data>

also problem occurs on diffrent tables on diffrent databases each time. Those are always corruptions in indexes I think, and always tables has at least 3 milions records.

Hi!

Since you use Percona Server, I'd suggest reporting the problem to Percona.
You could also create an instance based on an Official Oracle MySQL 5.5.32 or 5.6.12, created by importing a mysqldump to ensure a corruption doesn't exist in binary form.

Had you checked the health of the SSD?

"me too"

server version 5.5.32

InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 170438.
InnoDB: You may have to recover from a backup.
[dump snipped]
InnoDB: End of page dump
130711 11:02:13  InnoDB: Page checksum 2678445019, prior-to-4.0.14-form checksum 1082141312
InnoDB: stored checksum 2579455931, prior-to-4.0.14-form stored checksum 1082141312
InnoDB: Page lsn 2 384641961, low 4 bytes of lsn at page end 384641961
InnoDB: Page number (if stored to page already) 170438,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 1055
etc.

The same database fell over yesterday in the same way, albeit on a different table.
(I recovered by dumping database, dropping it and recovering from mysqldump file).

Backing store is a mirror of two ST1000DM003-1CH162 CC46 - no drive errors and no mirror errors.

Some table info:

mysql> show table status where Name like "%Incoming%";
+--------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+---------+
| Name               | Engine | Version | Row_format | Rows    | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free  | Auto_increment | Create_time         | Update_time | Check_time | Collation         | Checksum | Create_options | Comment |
+--------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+---------+
| TicketIncomingLogs | InnoDB |      10 | Compact    | 5809295 |            104 |   607125504 |               0 |            0 | 1842348032 |           NULL | 2013-07-10 07:35:26 | NULL        | NULL       | latin1_swedish_ci |     NULL |                |         |
+--------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+---------+

mysql> show table status where Name like "Messages";
+----------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+---------+
| Name     | Engine | Version | Row_format | Rows  | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free  | Auto_increment | Create_time         | Update_time | Check_time | Collation         | Checksum | Create_options | Comment |
+----------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+---------+
| Messages | InnoDB |      10 | Compact    | 76523 |           2009 |   153747456 |               0 |     30572544 | 1842348032 |         745141 | 2013-07-10 07:41:14 | NULL        | NULL       | latin1_swedish_ci |     NULL |                |         |
+----------+--------+---------+------------+-------+----------------+-------------+-----------------+--------------+------------+----------------+---------------------+-------------+------------+-------------------+----------+----------------+---------+

Have you solved this issue? 

I've faced the same with the latest 5.6.35-80.