Bug #50403 MySQL crashed not able to recover (InnoDB)
Submitted: 18 Jan 2010 6:10 Modified: 23 Aug 2010 15:09
Reporter: Meghanand Acharekar Email Updates:
Status: No Feedback Impact on me:
None 
Category:MySQL Server: InnoDB storage engine Severity:S1 (Critical)
Version:5.1.41 OS:Linux
Assigned to: CPU Architecture:Any
Tags: MySql Crash

[18 Jan 2010 6:10] Meghanand Acharekar
Description:
Not able to start MySQL (Using mysqld_safe),
MySQL 5.1 Crashed which was working fine,Crashed due to some Unknow reason

Following error
Error Log
----------
<[ Lots of Zeros ]>

100118 10:36:36  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
100118 10:36:36InnoDB: Error: trying to access a stray pointer 0x35bfbff8
InnoDB: buf pool start is at 0xb5bec000, end at 0xb63ec000
InnoDB: Probable reason is database corruption or memory
InnoDB: corruption. If this happens in an InnoDB database recovery, see
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: how to force recovery.
100118 10:36:36  InnoDB: Assertion failure in thread 3086673616 in file ../../storage/innobase/include/buf0buf.ic line 264
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
100118 10:36:36 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=500
threads_connected=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 403540 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x30000
/usr/local/mysql/libexec/mysqld(my_print_stacktrace+0x22) [0x847e032]
/usr/local/mysql/libexec/mysqld(handle_segfault+0x37f) [0x81c511f]
[0xb4f420]
/lib/i686/nosegneg/libc.so.6(abort+0x101) [0x12bba1]
/usr/local/mysql/libexec/mysqld(page_cur_search_with_match+0x503) [0x83b8d23]
/usr/local/mysql/libexec/mysqld(btr_cur_search_to_nth_level+0x593) [0x8356383]
/usr/local/mysql/libexec/mysqld(btr_pcur_open_on_user_rec+0x6b) [0x840b72b]
/usr/local/mysql/libexec/mysqld [0x8377031]
/usr/local/mysql/libexec/mysqld(dict_load_sys_table+0x68) [0x8378388]
/usr/local/mysql/libexec/mysqld(dict_boot+0xc1e) [0x840d93e]
/usr/local/mysql/libexec/mysqld(innobase_start_or_create_for_mysql+0x1228) [0x83e83e8]
/usr/local/mysql/libexec/mysqld [0x83442d0]
/usr/local/mysql/libexec/mysqld(ha_initialize_handlerton(st_plugin_int*)+0x34) [0x82b2e34]
/usr/local/mysql/libexec/mysqld [0x832dcbe]
/usr/local/mysql/libexec/mysqld(plugin_init(int*, char**, int)+0x8d0) [0x83302c0]
/usr/local/mysql/libexec/mysqld [0x81c6684]
/usr/local/mysql/libexec/mysqld(main+0x495) [0x81c9b95]
/lib/i686/nosegneg/libc.so.6(__libc_start_main+0xe0) [0x116f70]
/usr/local/mysql/libexec/mysqld [0x810a861]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
100118 10:36:36 mysqld_safe mysqld from pid file /usr/local/mysql/var/Devserver2.pid ended
100118 10:48:57 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/var
100118 10:48:57 [ERROR] You need to use --log-bin to make --binlog-format work.
100118 10:48:57 [ERROR] Aborting

100118 10:48:57 [Note] /usr/local/mysql/libexec/mysqld: Shutdown complete

100118 10:48:57 mysqld_safe mysqld from pid file /usr/local/mysql/var/Devserver2.pid ended
100118 10:56:08 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/var
100118 10:56:08 [ERROR] You need to use --log-bin to make --binlog-format work.
100118 10:56:08 [ERROR] Aborting

100118 10:56:08 [Note] /usr/local/mysql/libexec/mysqld: Shutdown complete

100118 10:56:08 mysqld_safe mysqld from pid file /usr/local/mysql/var/Devserver2.pid ended

How to repeat:
Unsure about this,may happened due to heavy load.

Suggested fix:
I have tried some tricks suggested at following pages.

http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
http://dev.mysql.com/doc/refman/5.1/en/crashing.html

but have no success.
[19 Jan 2010 8:12] Sveta Smirnova
Thank you for the report.

Error log shows your InnoDB tablespace is badly corrupted. In all similar cases reported before was because tablespace was rewritten not by mysqld process. Can it be same thing?

Did you try forcing InnoDB recovery? See http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html for details.
[19 Jan 2010 14:33] Meghanand Acharekar
Yes,
I tried this but no luck
Getting same error
[19 Jan 2010 18:20] Valeriy Kravchuk
What exact version, 5.1.x, do you use? Check bug #44416 for a similar crash...
[20 Jan 2010 6:16] Meghanand Acharekar
MySQL 5.1.41
[23 Jul 2010 15:09] Sveta Smirnova
Thank you for the feedback.

Can your ibdata file be corrupted by anything outside of mysqld process? Have you checked hard disk for errors?
[23 Aug 2010 23:00] Bugs System
No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".