Bug #77180 InnoDB get corrupted during a SELECT query cause by mydumper
Submitted: 28 May 2015 8:26 Modified: 15 Jul 2016 13:58
Reporter: Paul Namuag Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S2 (Serious)
Version:5.5.40 OS:Linux (CentOS release 6.5 (Final))
Assigned to: CPU Architecture:Any
Tags: corruption, innodb

[28 May 2015 8:26] Paul Namuag
Description:
This occurred when a mydumper process is running doing a SELECT query. This occurs mostly like most of the time during a month however, problem is that based on the error logs, after a restart server went OK but without LSN issues. This time, it shows that system LSN is behind of InnoDB page log.

150527 21:11:04  InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 219:3500760, should be 219:3531670!
150527 21:11:04  InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 219:3500761, should be 219:3531671!
150527 21:11:04  InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 219:3500762, should be 219:3531672!
150527 21:11:04  InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 219:3500763, should be 219:3531673!
150527 21:11:04  InnoDB: Assertion failure in thread 140264765363968 in file btr0pcur.c line 430
InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == buf_block_get_page_no(btr_pcur_get_block(cursor))
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
01:11:04 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=83
max_threads=3000
thread_count=20
connection_count=20
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6595338 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x100bbca0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f91ef838d98 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x78c4ce]
/usr/libexec/mysqld(handle_fatal_signal+0x493)[0x674fe3]
/lib64/libpthread.so.0(+0xf710)[0x7f942f783710]
/lib64/libc.so.6(gsignal+0x35)[0x7f942dbcc635]
/lib64/libc.so.6(abort+0x175)[0x7f942dbcde15]
/usr/libexec/mysqld[0x863e6b]
/usr/libexec/mysqld[0x81d524]
/usr/libexec/mysqld[0x81dce0]
/usr/libexec/mysqld[0x7fdeec]
/usr/libexec/mysqld(_ZN7handler15read_range_nextEv+0x2b)[0x67661b]
/usr/libexec/mysqld(_ZN7handler21read_multi_range_nextEPP18st_key_multi_range+0x24)[0x675924]
/usr/libexec/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x3d)[0x71fbad]
/usr/libexec/mysqld[0x7347d1]
/usr/libexec/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x71)[0x5aabe1]
/usr/libexec/mysqld[0x5ab4f7]
/usr/libexec/mysqld(_ZN4JOIN4execEv+0xbde)[0x5bd26e]
/usr/libexec/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x162)[0x5be8d2]
/usr/libexec/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x174)[0x5bf354]
/usr/libexec/mysqld[0x57ead7]
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x205b)[0x5842cb]
/usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x19d)[0x5869ed]
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1c0e)[0x588dae]
/usr/libexec/mysqld(_Z24do_handle_one_connectionP3THD+0xd2)[0x619b92]
/usr/libexec/mysqld(handle_one_connection+0x50)[0x619ca0]
/lib64/libpthread.so.0(+0x79d1)[0x7f942f77b9d1]
/lib64/libc.so.6(clone+0x6d)[0x7f942dc8286d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f9190004b40): is an invalid pointer
Connection ID (thread ID): 2004175
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150527 21:11:05 mysqld_safe Number of processes running now: 0
150527 21:11:05 mysqld_safe mysqld restarted
150527 21:11:06 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
150527 21:11:06 [Warning] Using unique option prefix myisam_recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150527 21:11:06 [Note] Plugin 'FEDERATED' is disabled.
150527 21:11:06 InnoDB: The InnoDB memory heap is disabled
150527 21:11:06 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150527 21:11:06 InnoDB: Compressed tables use zlib 1.2.3
150527 21:11:06 InnoDB: Using Linux native AIO
150527 21:11:06 InnoDB: Initializing buffer pool, size = 8.0G
150527 21:11:07 InnoDB: Completed initialization of buffer pool
150527 21:11:07 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 4327367571782
150527 21:11:07  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 4327372814336
InnoDB: Doing recovery: scanned up to log sequence number 4327378057216
InnoDB: Doing recovery: scanned up to log sequence number 4327383300096
InnoDB: Doing recovery: scanned up to log sequence number 4327388542976
InnoDB: Doing recovery: scanned up to log sequence number 4327393785856
InnoDB: Doing recovery: scanned up to log sequence number 4327399028736
InnoDB: Doing recovery: scanned up to log sequence number 4327404271616
...
...
...
150527 21:11:32  InnoDB: Error: page 7 log sequence number 4327865361447
InnoDB: is in the future! Current system log sequence number 4327578959847.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: for more information.
150527 21:11:32  InnoDB: Error: page 4 log sequence number 4327900146341
InnoDB: is in the future! Current system log sequence number 4327578959847.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: for more information.
150527 21:11:32  InnoDB: Error: page 5 log sequence number 4327900330742
InnoDB: is in the future! Current system log sequence number 4327578959847.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: for more information.

How to repeat:
N/A

Suggested fix:
Is this a bug or hardware failure? I don't see any problem yet on hardware stuff except the backup disk has disk issues
[28 May 2015 12:28] Valeriy Kravchuk
Had you used innodb_force_revocery=N where N > 3 on this instance before?

Check https://bugs.mysql.com/bug.php?id=71420
[2 Jun 2015 12:05] MySQL Verification Team
Thank you for the report.
Please see Bug #71420 as Valeriy pointed out in earlier note.

Also, see Bug #71420 -  20 Jan 2014 8:16] Shane Bester

yes, anything beyond 3 can permanently corrupt the database
4 can corrupt secondary indexes
5 can cause inconsistent results, also corrupt sec indexes (even if 4 did not have any effect)
[3 Jul 2015 1:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".
[23 Aug 2015 15:27] Valeriy Kravchuk
Users have similar assertion failures with versions 5.1.x as well...