Bug #71957 Repeated crash "InnoDB: Database page corruption on disk or a failed"
Submitted: 6 Mar 2014 11:45 Modified: 15 Jan 2015 19:30
Reporter: Aparimana Hone Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:5.5.35-0+wheezy1 (Debian) OS:Linux (Debian 7)
Assigned to: CPU Architecture:Any
Tags: corruption, crash, innodb, page corruption

[6 Mar 2014 11:45] Aparimana Hone
Description:
InnoDB keeps crashing (every few hours on a not-very-busy system) reporting page corruption.

This initially happened on version 5.5.31-0+wheezy1 ... after an upgrade to 5.5.35, it continues.  

Sometimes the system auto-recovers, sometimes it takes down the server and manual recovery is required (dump and recreate).

Any of the innodb databases can appear to be the cause.

There is no indication of actual hardware failure on the system:
- fairly new, high quality Raid 10 drives
- fsck reveals no errors
- no other part of the system is suffering errors

Typical error log file contents below, where auto-recovery worked (dump truncated):

InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 42958.
InnoDB: You may have to recover from a backup.
140306 10:15:27  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 000000000000000000000000000000000000000000000000000000000000000000<snip>
InnoDB: End of page dump
140306 10:15:27  InnoDB: Page checksum 3395813212, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 2050666811
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 1028229473
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 42958.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
140306 10:15:27  InnoDB: Assertion failure in thread 140183278556928 in file buf0buf.c line 3619
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:15:27 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=9
max_threads=151
thread_count=5
connection_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346700 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f7efaf8d260
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f7ef6858e80 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f7ef6f30e59]
/usr/sbin/mysqld(handle_fatal_signal+0x3d8)[0x7f7ef6e18808]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)[0x7f7ef65c7030]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f7ef4e59475]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7f7ef4e5c6f0]
/usr/sbin/mysqld(+0x5c9589)[0x7f7ef6fbf589]
/usr/sbin/mysqld(+0x5d415c)[0x7f7ef6fca15c]
/usr/sbin/mysqld(+0x5d4acc)[0x7f7ef6fcaacc]
/usr/sbin/mysqld(+0x5c55a8)[0x7f7ef6fbb5a8]
/usr/sbin/mysqld(+0x5a7559)[0x7f7ef6f9d559]
/usr/sbin/mysqld(+0x5a7cb9)[0x7f7ef6f9dcb9]
/usr/sbin/mysqld(+0x5a8783)[0x7f7ef6f9e783]
/usr/sbin/mysqld(+0x5e27a7)[0x7f7ef6fd87a7]
/usr/sbin/mysqld(+0x55c731)[0x7f7ef6f52731]
/usr/sbin/mysqld(+0x36ea10)[0x7f7ef6d64a10]
/usr/sbin/mysqld(+0x36e423)[0x7f7ef6d64423]
/usr/sbin/mysqld(_Z14get_all_tablesP3THDP10TABLE_LISTP4Item+0x9d3)[0x7f7ef6d73db3]
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x218)[0x7f7ef6d748c8]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x4ed)[0x7f7ef6d5df6d]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11d)[0x7f7ef6d59d4d]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x174)[0x7f7ef6d5fe14]
/usr/sbin/mysqld(+0x322064)[0x7f7ef6d18064]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x133a)[0x7f7ef6d1edfa]
/usr/sbin/mysqld(+0x32c5be)[0x7f7ef6d225be]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x184e)[0x7f7ef6d2466e]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x105)[0x7f7ef6dbf765]
/usr/sbin/mysqld(handle_one_connection+0x50)[0x7f7ef6dbf880]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f7ef65beb50]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f7ef4f030ed]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f7edc5b6190): is an invalid pointer
Connection ID (thread ID): 180
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
140306 10:15:27 mysqld_safe Number of processes running now: 0
140306 10:15:27 mysqld_safe mysqld restarted
140306 10:15:28 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
140306 10:15:28 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140306 10:15:28 [Note] Plugin 'FEDERATED' is disabled.
140306 10:15:28 InnoDB: The InnoDB memory heap is disabled
140306 10:15:28 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140306 10:15:28 InnoDB: Compressed tables use zlib 1.2.7
140306 10:15:28 InnoDB: Using Linux native AIO
140306 10:15:28 InnoDB: Initializing buffer pool, size = 128.0M
140306 10:15:28 InnoDB: Completed initialization of buffer pool
140306 10:15:28 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 64183964045
140306 10:15:28  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 64184113833
140306 10:15:30  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
140306 10:15:31  InnoDB: Waiting for the background threads to start
140306 10:15:32 InnoDB: 5.5.35 started; log sequence number 64184113833
140306 10:15:32 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
140306 10:15:32 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
140306 10:15:32 [Note] Server socket created on IP: '127.0.0.1'.
140306 10:15:32 [Note] Event Scheduler: Loaded 0 events
140306 10:15:32 [Note] /usr/sbin/mysqld: ready for connections.

How to repeat:
Sorry, I do not know how to reproduce this error, though it has been happening every few hours for the last few days.
[22 Jul 2014 19:31] Sveta Smirnova
Thank you for the report.

Please try to identify if this is same query which causes the crash. For example, you can turn general query log ON and check its content at the crash time.
[23 Jul 2014 11:53] Cristian Datculescu
Hello. The same issue affects me. We are running developer VM's and from time to time mysql crashes with the same symptoms. The machines are debian 6 with mysql 5.1.73 installed on them.

Most times the issue happens when dealing with large databases. The big issue is that it happens on different machines, and it happens randomly for random queries (never the same query twice). Sometimes selects, sometimes inserts/updates.
[28 Oct 2014 21:21] Ramon de Groot
this issue started affecting me since about a week. I am on ubuntu 14.04 and mysql --version gives:
mysql  Ver 14.14 Distrib 5.6.19, for debian-linux-gnu (x86_64) using  EditLine wrapper
I have checked my disk using smartctl and no issues came up. if it helps I can post the mysql error.log. it is very similar to the one the original reporter posted. the bug has happened during reads and inserts on 2 different tables so far.
[8 Dec 2014 18:27] Sveta Smirnova
Thank you for the feedback.

Ramon, yes, please, send us error log file. Please also send output of SHOW CREATE TABLE, SHOW TABLE STATUS for both affected tables and, if possible, try to get core file next time mysqld crashes and send us it together with mysqld binary you use.
[9 Dec 2014 21:35] Ramon de Groot
it turned out that my problems were caused by a defective RAM stick. after removing the mysql issue (and others, but it started manifesting itself in mysql) went away
[15 Dec 2014 19:30] Sveta Smirnova
Ramon,

thank you for the feedback.

Other reporters, please, try to get core file next time mysqld crashes and send us it together with mysqld binary you use and with error log file.
[16 Jan 2015 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".