Bug #78365 InnoDB: Assertion failure in thread ... in file trx0trx.cc line 292
Submitted: 8 Sep 2015 11:49 Modified: 16 Sep 2015 7:25
Reporter: Van Stokes Email Updates:
Status: Analyzing Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S1 (Critical)
Version:5.6.26-log OS:Ubuntu (14.04.03 x64)
Assigned to: Umesh Shastry CPU Architecture:Any
Tags: assertion, innodb, MySQL, server

[8 Sep 2015 11:49] Van Stokes
Description:
Server hung due to Intel network card bug in latest kernel:
https://bugs.launchpad.net/ubuntu/+source/linux-lts-utopic/+bug/1492146

Had to perform a cold start on the server and change kernel. Server is now stable however mysql will not start:

from the error.log (attached)

InnoDB: Last MySQL binlog file position 0 1102629, file name master-bin.001789
InnoDB: Cleaning up trx with id 13330155401
2015-09-08 07:34:51 7fa653199780  InnoDB: Assertion failure in thread 140352335484800 in file trx0trx.cc line 292
InnoDB: Failing assertion: trx->update_undo == NULL
InnoDB: We intentionally generate a memory trap.

Server cannot recover.

How to repeat:
Shutdown MySQL dirty - cold stop.

Suggested fix:
Not known.

We modified my.cnf 'skip-slave-start = 1'.
[8 Sep 2015 11:49] Van Stokes
Error log showing assertion information

Attachment: 21050908.error.mysql01.log (text/x-log), 5.12 KiB.

[8 Sep 2015 11:49] Van Stokes
The servers my.cnf file.

Attachment: mysql01.my.cnf (application/octet-stream, text), 8.62 KiB.

[8 Sep 2015 12:44] Van Stokes
Following: 
http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html

Had to do "innodb-force-recovery = 4" to get the server to start.

According to the documentation: "Values of 4 or greater can permanently corrupt data files." Therefore, we are considering the server as lost and we will rebuild from another master (we are using GTID).