Bug #78673 mysql won't start - has suffered Innodb crash
Submitted: 1 Oct 2015 15:05 Modified: 2 Oct 2015 10:56
Reporter: Steve Roberts Email Updates:
Status: Open Impact on me:
None 
Category:MySQL Server Severity:S1 (Critical)
Version: OS:Ubuntu
Assigned to: CPU Architecture:Any
Tags: can't repair index on innodb, crash, won't start

[1 Oct 2015 15:05] Steve Roberts
Description:
Server has crashed and won't start 
Have run  myisamchk  --force */*.MYI - looks clean
Error file says InnoDB is corrupt after running the repair
Server is a replicated slave.  Has been running fine.  Mostly SELECTs and insert/updates from the master

Table sizes are big - all tables are approx 200Gb

Disk space getting tight - 18Gb free.

151001 14:44:43 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will 
be removed in a future release. Please use the full name instead.
151001 14:44:43 [Note] Plugin 'FEDERATED' is disabled.
151001 14:44:43 InnoDB: The InnoDB memory heap is disabled
151001 14:44:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151001 14:44:43 InnoDB: Compressed tables use zlib 1.2.3.4
151001 14:44:43 InnoDB: Initializing buffer pool, size = 128.0M
151001 14:44:43 InnoDB: Completed initialization of buffer pool
151001 14:44:43 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 664967400878
151001 14:44:43  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 664967411595
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 7 row operations to undo
InnoDB: Trx id counter is 2AB32F00
151001 14:44:43  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 
37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78
 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: Starting in background the rollback of uncommitted transactions
151001 14:44:44  InnoDB: Rolling back trx with id 2AB32D00, 7 rows to undo
151001 14:44:44  InnoDB: Waiting for the background threads to start
151001 14:44:44  InnoDB: Assertion failure in thread 139917177534208 in file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
14:44:44 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346700 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f41198e5d79]
/usr/sbin/mysqld(handle_fatal_signal+0x483)[0x7f41197aa923]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f41184f2cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f4117b5e0d5]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f4117b6183b]
/usr/sbin/mysqld(+0x689e8f)[0x7f4119a13e8f]
/usr/sbin/mysqld(+0x60fae2)[0x7f4119999ae2]
/usr/sbin/mysqld(+0x624ab9)[0x7f41199aeab9]
/usr/sbin/mysqld(+0x61cc66)[0x7f41199a6c66]
/usr/sbin/mysqld(+0x61deb0)[0x7f41199a7eb0]
/usr/sbin/mysqld(+0x61781f)[0x7f41199a181f]
/usr/sbin/mysqld(+0x6cb350)[0x7f4119a55350]
/usr/sbin/mysqld(+0x6cb8ea)[0x7f4119a558ea]
/usr/sbin/mysqld(+0x617d8d)[0x7f41199a1d8d]
/usr/sbin/mysqld(+0x6183be)[0x7f41199a23be]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f41184eae9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f4117c1b8bd]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
151001 14:44:44 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will 
be removed in a future release. Please use the full name instead.
151001 14:44:44 [Note] Plugin 'FEDERATED' is disabled.
151001 14:44:44 InnoDB: The InnoDB memory heap is disabled
151001 14:44:44 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151001 14:44:44 InnoDB: Compressed tables use zlib 1.2.3.4
151001 14:44:44 InnoDB: Initializing buffer pool, size = 128.0M
151001 14:44:44 InnoDB: Completed initialization of buffer pool
151001 14:44:44 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 664967400878
151001 14:44:44  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 664967411595
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 7 row operations to undo
InnoDB: Trx id counter is 2AB32F00
151001 14:44:44  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 7
7 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: Starting in background the rollback of uncommitted transactions
151001 14:44:45  InnoDB: Rolling back trx with id 2AB32D00, 7 rows to undo
151001 14:44:45  InnoDB: Waiting for the background threads to start
151001 14:44:45  InnoDB: Assertion failure in thread 139744741881600 in file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
14:44:45 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

How to repeat:
Repeatable - server won't start
[2 Oct 2015 10:56] Steve Roberts
I managed to get the server to at least run with 

innodb_force_recovery = 3

Levels 1,2 fail

The issue is a corrupt index on innodb.
I can't drop the index - I get
ERROR 1031 (HY000): Table storage engine for 'directors' doesn't have this option

I can't change the table to MyISAM
mysql> ALTER TABLE directors ENGINE=myisam;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

Additionally I probably don't have enough spare disk space for the additional copy of the table.

Not sure why the tables are locked. I have run unlock tables.

I have stopped the slave. The slave error log shows:
               Last_SQL_Error: Error 'Lock wait timeout exceeded; try restarting transaction' on query. Default database: 'distressed_companies'. Query: 'DELETE FROM directors WHERE Company='NI0349''
Can't seem to clear the lock.