MySQL Bugs: #67010: Database page corruption on disk or a failed-after os reboot everything works ok

Bug #67010	Database page corruption on disk or a failed-after os reboot everything works ok
Submitted:	28 Sep 2012 17:55	Modified:	16 Feb 2015 16:39
Reporter:	Jan Z	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server	Severity:	S2 (Serious)
Version:	5.2.24	OS:	Linux (3.2.29)
Assigned to:		CPU Architecture:	Any

Description:
Mysql server crashed third time for me with this error. Uptime (os and mysql) was 10 days, 35 days, 5 days. I can't start mysql server again without rebooting server. It fills error log with hex dumps quickly filling ssd drive. After reboot everything works ok for 5-35 days or something.

120928 19:27:01 [Note] /var/mysql5/bin/mysqld: Normal shutdown

120928 19:27:01 [Note] Event Scheduler: Purging the queue. 0 events
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3253.
InnoDB: You may have to recover from a backup.
120928 19:27:01 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 21d63f6600000cb500004c460000390e0000007f8f46d43145bf0000000000000000000313f000e13d6885160bf9126c000000050000038b0000000054d748110000$
InnoDB: End of page dump
120928 19:27:01 InnoDB: Page checksum 1288290917, prior-to-4.0.14-form checksum 1035202961
InnoDB: stored checksum 567689062, prior-to-4.0.14-form stored checksum 1035202961
InnoDB: Page lsn 127 2403783729, low 4 bytes of lsn at page end 2403783729
InnoDB: Page number (if stored to page already) 3253,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 201712
InnoDB: Page may be an index page where index id is 427827
InnoDB: (index "word_id" of table "eltech_Vphp908"."phpbb_search_wordmatch")
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3253.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3253.
InnoDB: You may have to recover from a backup.
120928 19:27:02 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 21d63f6600000cb500004c460000390e0000007f8f46d43145bf0000000000000000000313f000e13d6885160bf9126c000000050000038b0000000054d748110000$
InnoDB: End of page dump
120928 19:27:02 InnoDB: Page checksum 1288290917, prior-to-4.0.14-form checksum 1035202961
InnoDB: stored checksum 567689062, prior-to-4.0.14-form stored checksum 1035202961
InnoDB: Page lsn 127 2403783729, low 4 bytes of lsn at page end 2403783729
InnoDB: Page number (if stored to page already) 3253,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 201712
InnoDB: Page may be an index page where index id is 427827
InnoDB: (index "word_id" of table "eltech_Vphp908"."phpbb_search_wordmatch")
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3253.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3253.
InnoDB: You may have to recover from a backup.
120928 19:27:03 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 21d63f6600000cb500004c460000390e0000007f8f46d43145bf0000000000000000000313f000e13d6885160bf9126c000000050000038b0000000054d748110000$
InnoDB: End of page dump
120928 19:27:03 InnoDB: Page checksum 1288290917, prior-to-4.0.14-form checksum 1035202961
InnoDB: stored checksum 567689062, prior-to-4.0.14-form stored checksum 1035202961
InnoDB: Page lsn 127 2403783729, low 4 bytes of lsn at page end 2403783729
InnoDB: Page number (if stored to page already) 3253,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 201712
InnoDB: Page may be an index page where index id is 427827
InnoDB: (index "word_id" of table "eltech_Vphp908"."phpbb_search_wordmatch")
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3253.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
120928 19:27:03 InnoDB: Starting shutdown...
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3253.
InnoDB: You may have to recover from a backup.
120928 19:27:04 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 21d63f6600000cb500004c460000390e0000007f8f46d43145bf0000000000000000000313f000e13d6885160bf9126c000000050000038b0000000054d748110000$
InnoDB: End of page dump
120928 19:27:04 InnoDB: Page checksum 1288290917, prior-to-4.0.14-form checksum 1035202961
InnoDB: stored checksum 567689062, prior-to-4.0.14-form stored checksum 1035202961
InnoDB: Page lsn 127 2403783729, low 4 bytes of lsn at page end 2403783729
InnoDB: Page number (if stored to page already) 3253,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 201712
InnoDB: Page may be an index page where index id is 427827
InnoDB: (index "word_id" of table "eltech_Vphp908"."phpbb_search_wordmatch")
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 3253.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page

How to repeat:
-

Thank you for the report.

But version 5.2.24 is very old and many bugs were fixed since. Please try with current version 5.5.28 and, if issue still exists, provide full error log file.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

encounter the same problem on 5.5.18.

Thank you for the feedback.

Please provide full error log file.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".

I got the same problem, there is a killed session for show processlist like

137375	 d3	 10.20.0.124:59712	 ng18_rzhao	 Killed	 2465227	 Sending data	 select ..........

The session was killed week ago, and the page dump still go into mysqld.log, cause it increase to 380G two times. Anytime, I just clean it with

cat /dev/null > /var/log/mysqld.log

But the session is marked killed and still there, nothing stop the page dump. 

Piece of readable error in mysqld.log like

InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 4484.
InnoDB: You may have to recover from a backup.
140711 13:13:21 InnoDB: Page dump in ascii and hex (16384 bytes):

The major part of mysqld.log are unreadable. 

Thanks

Thank you for the feedback.

This error means your InnoDB data is corrupted and you should fix it as described at http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html. Actually both cases look like you have issue with disk, not as MySQL bug.

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".