Bug #74924 mysql crash with a mysqldump or simply select
Submitted: 19 Nov 2014 11:17 Modified: 22 Aug 2015 11:41
Reporter: Renato Lopes Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S1 (Critical)
Version:5.1.73-1 OS:Linux (Debian 6 x64)
Assigned to: CPU Architecture:Any

[19 Nov 2014 11:17] Renato Lopes
Description:
when i set recovery to 6 and try to get dump for some tables i get the error below.

141119  8:39:26  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
InnoDB: Page directory corruption: supremum not pointed to
141119  8:39:26  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 000...(blank SPACE);InnoDB: End of page dump
141119  8:39:26  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
141119  8:39:26  InnoDB: Assertion failure in thread 140567607629568 in file ../../../storage/innobase/rem/rem0rec.c line 337
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:39:26 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=2097152
read_buffer_size=2097152
max_used_connections=1
max_threads=100
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 259091 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fd9dfe935f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fd87251ce88 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7fd9d1ae7049]
/usr/sbin/mysqld(handle_fatal_signal+0x483) [0x7fd9d18f9c53]
/lib/libpthread.so.0(+0xeff0) [0x7fd9d1044ff0]
/lib/libc.so.6(gsignal+0x35) [0x7fd9cfae71b5]
/lib/libc.so.6(abort+0x180) [0x7fd9cfae9fc0]
/usr/sbin/mysqld(rec_get_offsets_func+0xe0) [0x7fd9d1a28ae0]
/usr/sbin/mysqld(page_cur_search_with_match+0x2cd) [0x7fd9d1a1872d]
/usr/sbin/mysqld(btr_cur_search_to_nth_level+0x629) [0x7fd9d19b5839]
/usr/sbin/mysqld(row_search_for_mysql+0xc26) [0x7fd9d1a3bdf6]
/usr/sbin/mysqld(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int)+0x7c) [0x7fd9d19a622c]
/usr/sbin/mysqld(+0x3e97de) [0x7fd9d185b7de]
/usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x81) [0x7fd9d1856351]
/usr/sbin/mysqld(+0x3e4e3d) [0x7fd9d1856e3d]
/usr/sbin/mysqld(JOIN::exec()+0xbad) [0x7fd9d186c94d]
/usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x12e) [0x7fd9d18685ee]
/usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x174) [0x7fd9d186df74]
/usr/sbin/mysqld(+0x385e1a) [0x7fd9d17f7e1a]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x516) [0x7fd9d17fbe26]
/usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x3fb) [0x7fd9d180141b]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x964) [0x7fd9d1801d94]
/usr/sbin/mysqld(do_command(THD*)+0xea) [0x7fd9d1802f1a]
/usr/sbin/mysqld(handle_one_connection+0x236) [0x7fd9d17f48d6]
/lib/libpthread.so.0(+0x68ca) [0x7fd9d103c8ca]
/lib/libc.so.6(clone+0x6d) [0x7fd9cfb84b6d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fd9dfeed5d0): select * FROM history_uint WHERE itemid = '25046'
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
141119  8:39:27 [Note] Plugin 'FEDERATED' is disabled.
141119  8:39:27  InnoDB: Initializing buffer pool, size = 5.0G
141119  8:39:28  InnoDB: Completed initialization of buffer pool
InnoDB: The user has set SRV_FORCE_NO_LOG_REDO on
InnoDB: Skipping log redo
141119  8:39:28  InnoDB: Started; log sequence number 0 0
InnoDB: !!! innodb_force_recovery is set to 6 !!!
141119  8:39:28 [Note] Event Scheduler: Loaded 0 events
141119  8:39:28 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.73-1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)

How to repeat:
set recovery to 6 and try to get dump for some tables i get the error.
[19 Nov 2014 18:06] Miguel Solorzano
Have you upgraded to 5.1.73 from older version?. Thanks.
[19 Nov 2014 20:05] Renato Lopes
No, my database was created on this version.
[22 Aug 2015 11:41] Shane Bester
Setting innodb-force-recovery to 6 can permanently cause more damage on older versions of innodb.  I hope you have a backup.

https://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html

To debug a root cause of the corruption, we'd need entire mysql error log from the beginning, as well as study the system error logs and check disks/memory for problems.   Added to this, 5.1 probably has a few corruption bugs that are only fixed in modern versions (5.6, 5.7).