MySQL Bugs: #85871: InnoDB: Corruption of an index tree

Bug #85871	InnoDB: Corruption of an index tree
Submitted:	10 Apr 2017 3:56	Modified:	11 Apr 2017 14:10
Reporter:	Yanguang Hou	Email Updates:
Status:	Not a Bug	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S3 (Non-critical)
Version:	5.6.28	OS:	CentOS
Assigned to:		CPU Architecture:	Any
Tags:	corruption, INDEX, innodb

Description:
2017-04-09 16:55:07 7fdf58641700 InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex af52af170001ec6200021c700001d7a4000007928dbe25d945bf000000000000000000000ea900383f7480e2000000003f330001000200e0000000012f72edb700000000000000001e120000000000000000000000000000000000000000010002001d696e66696d756d0005000b000073757072656d756d2418000000100049e7a2a7e7b4a0e5a082e5ae98e696b9e69797e888b0e5ba97999c3a199265643936393066382d613236302d346238662d383065632d3661333936656535326236382418000000180049e7a2a7e7b4a0e5a082e5ae98e696b9e69797e888b0e5ba97999c3b247061633935616331622d346162612d343661372d616633382d6662383238366365353034302418000000200049e7a2a7e7b4a0e5a082e5ae98e696b9e69797e888b0e5ba97999c3b419031313831613636372d633363382d343431322d383638392d3332366330643038626566392418000400280049e7a2a7e7b4a0e5a082e5ae98e696b9e69797e888b0e5ba97999c3b421c65313934646238372d326362642d346636662d393035322d3135316637373363643131612418000000300049e7a2a7e7b4a0e5a082e5ae98e696b9e69797e888....
InnoDB: End of page dump
2017-04-09 16:55:07 7fdf58641700 InnoDB: uncompressed page, stored checksum in field1 2941431575, calculated checksums for field1: crc32 280369570, innodb 2418454706, none 3735928559, stored checksum in f
ield2 1938, calculated checksums for field2: crc32 280369570, innodb 2547512064, none 3735928559, page LSN 1938 2378048985, low 4 bytes of LSN at page end 2378048985, page number (if stored to page alread
y) 126050, space id (if created with >= MySQL-4.1.1 and stored already) 3753
InnoDB: Page may be an index page where index id is 7698
InnoDB: (index "idx_tr_rnick_created" of table "hades6"."trade_rate")
InnoDB: Dump of the parent page:
2017-04-09 16:55:07 7fdf58641700 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 1fb4ef67000040070001ea130000c0aa00000775e3b0845945bf000000000000000000000ea9003037dd80c90000000037a10001000100c7000000000000000000010000000000001e120000000000000000000000000000000000000000
010002001d696e66696d756d0006000b000073757072656d756d240900000011003ee799bee6809de79b9f9999ccb9b262653538386633362d376637372d343866392d613935302d6566643433633063323765320001cae6240900000019003ee799bee6809d
e79b9f999a851ed933316234323766652d666363622d346632322d396638622d3966656234313839643833390001db332412000000210047e799bee789a7e69e97e69797e888b0e5ba97999a76612f31353063613232342d613761632d346432322d62376630
2d6264616333626366363937640000bf3f241b000400290050e799bee8a786e79.....
InnoDB: End of page dump
2017-04-09 16:55:07 7fdf58641700 InnoDB: uncompressed page, stored checksum in field1 531951463, calculated checksums for field1: crc32 3277227083, innodb 3517705687, none 3735928559, stored checksum in field2 0, calculated checksums for field2: crc32 3277227083, innodb 14144210, none 3735928559, page LSN 1909 3819996249, low 4 bytes of LSN at page end 0, page number (if stored to page already) 16391, space id (if created with >= MySQL-4.1.1 and stored already) 3753
InnoDB: Page may be an index page where index id is 7698
InnoDB: (index "idx_tr_rnick_created" of table "hades6"."trade_rate")
InnoDB: Corruption of an index tree: table "hades6"."trade_rate", index "idx_tr_rnick_created",
InnoDB: father ptr page no 138352, child page no 126050
PHYSICAL RECORD: n_fields 3; compact format; info bits 0
 0: len 24; hex e7a2a7e7b4a0e5a082e5ae98e696b9e69797e888b0e5ba97; asc                         ;;
 1: len 5; hex 999c3a1992; asc   :  ;;
 2: len 30; hex 65643936393066382d613236302d346238662d383065632d366133393665; asc ed9690f8-a260-4b8f-80ec-6a396e; (total 36 bytes);
 n_owned: 0; heap_no: 2; next rec: 201
PHYSICAL RECORD: n_fields 4; compact format; info bits 0
 0: len 24; hex e7a2a7e7b4a0e5a082e5ae98e696b9e69797e888b0e5ba97; asc                         ;;
 1: len 5; hex 999c277a94; asc   'z ;;
 2: len 30; hex 32616532383636392d303161312d343461662d383064662d653339363665; asc 2ae28669-01a1-44af-80df-e3966e; (total 36 bytes);
 3: len 4; hex 00021c70; asc    p;;
 n_owned: 0; heap_no: 190; next rec: 13824
InnoDB: You should dump + drop + reimport the table to fix the
InnoDB: corruption. If the crash happens at the database startup, see
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html about
InnoDB: forcing recovery. Then dump + drop + reimport.
2017-04-09 16:55:07 7fdf58641700  InnoDB: Assertion failure in thread 140597237389056 in file btr0btr.cc line 1466
InnoDB: We intentionally generate a memory trap.

How to repeat:
don't know now.

Hi!

The error message that was issued by InnoDB storage engine is self-explanatory. You have got a hardware corruption. When retrieving a page, checksums clearly showed that corruption did not occur at the beginning or at the end of the InnoDB page, but at the middle of it.

We have encountered many reports of this kind from the inception of the InnoDB storage engine.

In 95 % of the cases, problem was in the hardware. Please, do provide us with feedback, so that we could proceed further.

Do you use ECC RAM modules, 2 bits checking 1 bit correcting ??? Do you use mirrored disks or disks with some redundancy for the parity checking , like RAID.

If not, can you analyze your system logs, which are readily available on Red Hat OS. Please inspect them all on any report of any kind of hardware problem, be it CPU, caches, RAM, disk controllers, disks , caches on controllers or disks etc ..... Check whether there were any reports on files related to warnings or errors. Check whether there were reports on running out of space ....

Do analyze them for the period of the last three months.

If you do not find anything, can you make a thorough hardware analysis. I guess that this CentOS is a fully stable version ????

Let me inform you also that sometimes, commodity RAM without ECC can have glitches, which change memory contents. Those glitches are rare and practically never repeat in the same addresses. Also, glitches are quite frequent with commodity disk controllers and hard drive.

Running an SQL server requires a very reliable computer. I am writing this message from the machine with ECC memory and mirrored disks.