MySQL Bugs: #65196: InnoDB: Assertion failure in thread ... in file btr0cur.c

Bug #65196	InnoDB: Assertion failure in thread ... in file btr0cur.c
Submitted:	3 May 2012 21:19	Modified:	7 May 2012 7:45
Reporter:	martin fuxa	Email Updates:
Status:	Unsupported	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S2 (Serious)
Version:	5.5.21	OS:	Linux (Centos 5.6)
Assigned to:		CPU Architecture:	Any

Description:
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Error in pages 1220160 and 258049 of index "PRIMARY" of table "x"."x"
InnoDB: broken FIL_PAGE_NEXT or FIL_PAGE_PREV links
120426  8:07:10  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 6d9a5...

InnoDB: End of page dump
120426  8:07:10  InnoDB: Page checksum 1838829962 (32bit_calc: 2985562315), prior-to-4.0.14-form checksum 4108579494
InnoDB: stored checksum 1838829962, prior-to-4.0.14-form stored checksum 4108579494
InnoDB: Page lsn 344 1470213567, low 4 bytes of lsn at page end 1470213567
InnoDB: Page number (if stored to page already) 1220160,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 14074
InnoDB: (index "PRIMARY" of table "x"."x")
120426  8:07:10  InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex c642e5...

InnoDB: End of page dump
120426  8:07:10  InnoDB: Page checksum 3326272848 (32bit_calc: 2178245854), prior-to-4.0.14-form checksum 568955370
InnoDB: stored checksum 3326272848, prior-to-4.0.14-form stored checksum 568955370
InnoDB: Page lsn 337 1831943029, low 4 bytes of lsn at page end 1831943029
InnoDB: Page number (if stored to page already) 258049,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 14074
InnoDB: (index "PRIMARY" of table "x"."x")
120427  0:40:46  InnoDB: Assertion failure in thread 1251584320 in file btr0cur.c line 314
InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
22:40:46 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

full log attached

How to repeat:
I dont know

Suggested fix:
I dont know

err log

Attachment: t3523.log (text/x-log), 10.42 KiB.

and again and again ...

with innodb_force_recovery 1 to 4 never ending
120427 16:13:52  InnoDB: Waiting for the background threads to start

my innodb conf
innodb_buffer_pool_size = 7000M
innodb_flush_log_at_trx_commit = 2
innodb_log_file_size = 64M
innodb_additional_mem_pool_size = 16M
innodb_old_blocks_time=1000

after slave recovery from master
show table status like 'x';
+----------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+
| Name     | Engine | Version | Row_format | Rows     | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time         | Update_time | Check_time | Collation       | Checksum | Create_options | Comment |
+----------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+
| xxxxxxxx | InnoDB |      10 | Compact    | 45395275 |             46 |  2101346304 |               0 |   1146093568 |  17825792 |           NULL | 2012-05-02 13:34:23 | NULL        | NULL       | utf8_general_ci |     NULL |                |         |
+----------+--------+---------+------------+----------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------+

Are you checked hard drive failure?. Thanks.

HDD error, I can't exclude because

mysql bin log
tune4fs -l /dev/md1 |grep -i error
Errors behavior:          Continue

mysql data
tune4fs -l /dev/md0 |grep -i error
Errors behavior:          Continue

but in linux log/messages isn't nothing.
In daily routine server run's CHECK TABLE MEDIUM and there was few errors (see below), but not at same table (and not same version of Mysql server, server is updated regulary after version release, but ofcourse still 5.5.x).

2012-04-15 Index ... contains 36528795 entries, should be 36528796. 
ENGINE=InnoDB 
2012-04-09 Index ... contains 31081166 entries, should be 31081167.
ENGINE=InnoDB 
2012-04-01 Found 5868265 keys of 5868266 
ENGINE=MyISAM 
2012-03-30 Found 186 keys of 185 
ENGINE=MyISAM 
2012-03-25 Unexpected byte: 0 at link: 1333080144 
ENGINE=MyISAM

Thank you for the feedback.

What CHECK TABLE x EXTENDED returns?

Martin, since you're using a percona build, you should file a bug report there, or ask in their forums.  We can only support official Oracle MySQL builds here.

https://bugs.launchpad.net/percona-server/+bug/995258