MySQL Bugs: #86185: mysqld not starting on Production instance

Bug #86185	mysqld not starting on Production instance
Submitted:	4 May 2017 18:36	Modified:	5 May 2017 8:52
Reporter:	Sierra Atlantic	Email Updates:
Status:	Unsupported	Impact on me:	None
Category:	MySQL Server	Severity:	S1 (Critical)
Version:	5.0.95	OS:	Red Hat (Red Hat Enterprise Linux Server release 5.9 )
Assigned to:		CPU Architecture:	Any

Description:
170504 13:19:19 mysqld started
InnoDB: Log scan progressed past the checkpoint lsn 31 132645931
170504 13:19:20 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 31 132649112
InnoDB: Page directory corruption: supremum not pointed to
170504 13:19:20 InnoDB: Page dump in ascii and hex (16384 bytes):

170504 13:19:20 InnoDB: Page checksum 3318262733, prior-to-4.0.14-form checksum 3312935585
InnoDB: stored checksum 3318262733, prior-to-4.0.14-form stored checksum 3312935585
InnoDB: Page lsn 31 132643993, low 4 bytes of lsn at page end 132643993
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
170504 13:19:20InnoDB: Error: trying to access a stray pointer 0x80002abd0015bff8
InnoDB: buf pool start is at 0x2abd0014c000, end at 0x2ac00014c000
InnoDB: Probable reason is database corruption or memory
InnoDB: corruption. If this happens in an InnoDB database recovery, see
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: how to force recovery.
170504 13:19:20InnoDB: Assertion failure in thread 46991177624688 in file ./../include/buf0buf.ic line 268
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: about forcing recovery.
170504 13:19:20 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_connections=100
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 217600 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
frame pointer is NULL, did you compile with
-fomit-frame-pointer? Aborting backtrace!
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170504 13:19:21 mysqld ended

How to repeat:
N/A

Suggested fix:
N/A

We tried to for a force recovery by following the steps listed in
https://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
 but from the logs its doesn't seem to be kicking in.

[root@srv-1oscardb01 mysql]#  myisamchk --silent --force */*.MYI
myisamchk: MyISAM file oscar-pci/ccdetail.MYI
myisamchk: warning: 2 clients are using or haven't closed the table properly
myisamchk: MyISAM file primedia/osc_back_issue_tracking.MYI
myisamchk: warning: 1 client is using or hasn't closed the table properly
myisamchk: MyISAM file primedia/osc_ipad_adobe_device_tracking.MYI
myisamchk: warning: 2 clients are using or haven't closed the table properly

myisamchk: MyISAM file primedia/osc_mktg_offer_impressions.MYI
myisamchk: warning: 2 clients are using or haven't closed the table properly

myisamchk: MyISAM file primedia/osc_offer_impressions.MYI
myisamchk: warning: 2 clients are using or haven't closed the table properly

Thank you for taking the time to report a problem.  Unfortunately you are not using a current version of the product you reported a problem with -- the problem might already be fixed. 5.0 and 5.1 are both end-of-life (EOL) products. We won't spend time on this. Please download a new version from http://www.mysql.com/downloads/

Also, there is no test case provided in the bug report and hence there is nothing we can verify here.  If you are able to reproduce the bug with one of the latest versions, please attach the exact reproducible test case and change the version on this bug report to the version you tested and change the status back to "Open".  Again, thank you for your continued support of MySQL.