Bug #60888 mysqld on startup fails to recover
Submitted: 15 Apr 2011 22:47 Modified: 29 Apr 2011 17:38
Reporter: Guillaume Royer Email Updates:
Status: Can't repeat Impact on me:
None 
Category:MySQL Server: Errors Severity:S1 (Critical)
Version:5.1.26 OS:Linux
Assigned to: CPU Architecture:Any
Tags: receovery crash

[15 Apr 2011 22:47] Guillaume Royer
Description:
My mysql crashed at some point when I tried to change the root password. Since then, when I try to start it again, I have the following error in the log:

110415 18:29:52 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql-5.1.26/data
InnoDB: Log scan progressed past the checkpoint lsn 34 164654031
110415 18:29:52  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 34 164659897
110415 18:29:52  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 1 InnoDB: Probable data corruption on page 3366400
InnoDB: Original record PHYSICAL RECORD: n_fields 184; 1-byte offsets; info bits 0
 0: len 0; hex ; asc ;; 1: len 0; hex ; asc ;; 2: len[... Omitted hexadecimal code here...]

110415 18:29:52  InnoDB: Page checksum 3614028634, prior-to-4.0.14-form checksum 3073716135
InnoDB: stored checksum 3614028634, prior-to-4.0.14-form stored checksum 3073716135
InnoDB: Page lsn 0 2878842, low 4 bytes of lsn at page end 2878842
InnoDB: Page number (if stored to page already) 3366400,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 4294967295 0
InnoDB: (index "CLUST_IND" of table "SYS_IBUF_TABLE_0")
110415 18:29:52  InnoDB: Assertion failure in thread 3086289808 in file page/page0page.c line 133
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
/usr/local/mysql-5.1.26/bin/mysqld(print_stacktrace+0x1c)[0x82d26a8]
/usr/local/mysql-5.1.26/bin/mysqld(handle_segfault+0x1ce)[0x81c9f32]
[0xc62420]
/lib/libc.so.6(abort+0x101)[0x1e5691]
/usr/local/mysql-5.1.26/bin/mysqld(page_dir_find_owner_slot+0x19a)[0x837ab8e]
/usr/local/mysql-5.1.26/bin/mysqld(page_cur_delete_rec+0x52)[0x837963a]
/usr/local/mysql-5.1.26/bin/mysqld(page_cur_parse_delete_rec+0x9d)[0x8379581]
/usr/local/mysql-5.1.26/bin/mysqld[0x836f1de]
/usr/local/mysql-5.1.26/bin/mysqld(recv_recover_page+0x449)[0x836b689]
/usr/local/mysql-5.1.26/bin/mysqld(buf_page_io_complete+0x35c)[0x832da64]
/usr/local/mysql-5.1.26/bin/mysqld(fil_aio_wait+0xf0)[0x8348414]
/usr/local/mysql-5.1.26/bin/mysqld[0x83a2041]
/lib/libpthread.so.0[0x30c49b]
/lib/libc.so.6(clone+0x5e)[0x28c42e]
110415 18:29:52 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=8388608
max_used_connections=0
max_threads=151
threads_connected=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2737079 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
110415 18:29:52 mysqld_safe mysqld from pid file /usr/local/mysql-5.1.26/data/mysqld.pid ended

Since I cannot run mysqld, I cannot dump the databases, so I really need to find a solution to recover from this crash or lots of data may be lost!

How to repeat:
running mysqld start
[16 Apr 2011 2:17] MySQL Verification Team
Thank you for the bug report. Your version 5.2.26 is quite older the last release is 5.1.56 could you please upgrade and test.

http://dev.mysql.com/downloads/mysql/5.1.html#downloads

Thanks in advance.
[16 Apr 2011 3:29] Guillaume Royer
Thanks for your answer.

How can I upgrade in my case without loosing any data? my sql server was installed from a binary version, do I need to create a new directory then copy the content in the new one? Thanks
[28 Apr 2011 17:27] Sveta Smirnova
Thank you for the feedback.

Have you tried to force InnoDB recovery as described at http://dev.mysql.com/doc/refman/5.0/en/forcing-innodb-recovery.html?
[28 Apr 2011 22:17] Guillaume Royer
Thanks for your feedback,
After upgrading to the latest version, it managed to recover the databases automatically.
[29 Apr 2011 17:34] Sveta Smirnova
Thank you for the feedback.

Closed as "Can't repeat" as it is not repeatable anymore.