MySQL Bugs: #83058: InnoDB: Assertion failure in thread

Bug #83058	InnoDB: Assertion failure in thread
Submitted:	20 Sep 2016 11:08	Modified:	7 Mar 2017 15:28
Reporter:	Andrea Dante Bozzola	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S1 (Critical)
Version:	5.7.15 Commercial	OS:	CentOS (6.7)
Assigned to:		CPU Architecture:	Any
Tags:	5.7, assertion, commercial, innodb

Description:
MySQL server crashes as soon as it starts.

innodb_buffer_pool_size is 78G and total server memory is 94G so I exclude a memory problem; the server actually replicates in a master slave configuration, but replication had not been started yet so I exclude it could be caused by that either.

Follows an extraction of the error log file:
=============================================================================
2016-09-20T08:44:56.233243Z 0 [Note] InnoDB: 1 transactions in prepared state after recovery
2016-09-20T08:44:56.233248Z 0 [Note] Found 1 prepared transaction(s) in InnoDB
2016-09-20 09:44:56 0x7f5651743720  InnoDB: Assertion failure in thread 140008710485792 in file fut0lst.ic line 85
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
08:44:56 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=2048
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 830144 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf54f55]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7deff4]
/lib64/libpthread.so.0(+0xf7e0)[0x7f56513257e0]
/lib64/libc.so.6(gsignal+0x35)[0x7f564fdc55e5]
/lib64/libc.so.6(abort+0x175)[0x7f564fdc6dc5]
/usr/sbin/mysqld(_Z18ut_print_timestampP8_IO_FILE+0x0)[0x7cda84]
/usr/sbin/mysqld(_Z18flst_insert_beforePhS_S_P5mtr_t+0x150)[0x122fda0]
/usr/sbin/mysqld(_Z14flst_add_firstPhS_P5mtr_t+0x1b0)[0x12302c0]
/usr/sbin/mysqld(_Z36trx_purge_add_update_undo_to_historyP5trx_tP14trx_undo_ptr_tPhbmP5mtr_t+0x154)[0x11326a4]
/usr/sbin/mysqld(_Z23trx_undo_update_cleanupP5trx_tP14trx_undo_ptr_tPhbmP5mtr_t+0x1f)[0x115c05f]
/usr/sbin/mysqld(_Z31trx_write_serialisation_historyP5trx_tP5mtr_t+0x361)[0x1153831]
/usr/sbin/mysqld(_Z14trx_commit_lowP5trx_tP5mtr_t+0x49)[0x1156c29]
/usr/sbin/mysqld(_Z10trx_commitP5trx_t+0x119)[0x11575e9]
/usr/sbin/mysqld(_Z20trx_commit_for_mysqlP5trx_t+0xef)[0x1157d0f]
/usr/sbin/mysqld(_Z19innobase_commit_lowP5trx_t+0x17)[0xff4827]
/usr/sbin/mysqld[0x1002340]
/usr/sbin/mysqld[0xde6e77]
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x1b9)[0xd41419]
/usr/sbin/mysqld(_Z10ha_recoverP7st_hash+0xba)[0xde5f5a]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG7recoverEP11st_io_cacheP28Format_description_log_eventPy+0x18f)[0xeeb7ff]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG11open_binlogEPKc+0x487)[0xef87d7]
/usr/sbin/mysqld[0x7d5c9d]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x53d)[0x7d88ad]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f564fdb1d1d]
/usr/sbin/mysqld[0x7cee39]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2016-09-20T08:44:56.432800Z mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
=============================================================================

I intend to check if the same issue arises with a downgraded version (5.7.10)

How to repeat:
No

Suggested fix:
No

Hi,

When you get an assertion failure on the recovery, that means that something went very much wrong during the last shutdown.

Can you describe the shutting down of the MySQL server, before this recovery started ??? Is it possible that shutdown did not proceed properly ??? Was there a crash or power problems. 

When was the last time that you checked your hard disk or SSD, what ever you are using ??? Are you using a RAID, so that when your disk gets bad segments, new one is generated from the redundant data ???

Do you have ECC RAM modules, two bits checking and one bit correcting ??? If not, it is highly advisable to check for the RAM modules sanity. I must add that some hardware, especially the commodity ones, can develop RAM glitches which are only temporary and rare.

Do you have UPS or some other way of protection from power failures or spikes ???

It is also highly desirable that you upload to this bug your entire error log, so that we have a report of what kind of the shutdown has occurred prior to this recovery process ??

We would also like to have copies of the system logs, for the period that would cover both the last shutdown and this recovery.

Thank you very much, in advance ....

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".