MySQL Bugs: #65297: InnoDB ignores a possible problem when decompressing blobs

Bug #65297	InnoDB ignores a possible problem when decompressing blobs
Submitted:	12 May 2012 19:08	Modified:	13 Jun 2012 7:46
Reporter:	Nizameddin Ordulu	Email Updates:
Status:	No Feedback	Impact on me:	None
Category:	MySQL Server: InnoDB Plugin storage engine	Severity:	S2 (Serious)
Version:		OS:	Any
Assigned to:		CPU Architecture:	Any
Tags:	compression, innodb

Description:
We recently modified the zlib parameters InnoDB uses for better compression and not writing adler32 checksums when producing the compression stream. During the decompression we first try to decompress assuming adler32 checksums exist. zlib pretty quickly understands whether the checksum exists because it's in the first compressed block. If the first call to inflate() fails using the checksum then it means that the data was compressed without the checksums and we reset the stream and decompress without the checksums. This is done to have backwards compatibility with the existing data.

Recently when we deployed the new RPM with the above changes we started seeing messages like this in the mysql logs: 
InnoDB: inflate() of compressed BLOB page 248 space 9371 returned 1 ((null))

The message is printed in btr0cur.c:
4798    default:
4799inflate_error:
4800      ut_print_timestamp(stderr);
4801      fprintf(stderr,
4802        "  InnoDB: inflate() of"
4803        " compressed BLOB"
4804        " page %lu space %lu returned %d (%s)\n",
4805        (ulong) page_no, (ulong) space_id,
4806        err, d_stream->msg);
4807    case Z_BUF_ERROR:
4808      goto end_of_blob;
4809    }

It seems like the problem is that the zlib thinks that the compression stream has ended but the pointers in blob headers have a non-null next page so innodb thinks the blob has more data on the next blob page. Even though a message is logged, the error seems to be ignored on purpose. Can you explain the reason why you decided to ignore this error? This may potentially return only part of a blob if zlib was wrong and the stream did not end. I don't think it's the case for us, I think the blob pointers were wrong but I would like to know what you think.

thanks,
nizam

How to repeat:
read code.

Ignoring BLOB read errors is in line with the existing code for uncompressed tables. It is admittedly bad practice. At that time I thought that the existing code was like that on purpose, to allow as complete data recovery as possible when using something innodb_force_recovery

Since the initial release of InnoDB Plugin for MySQL 5.1, we have found and fixed many BLOB bugs. I now think that the BLOB code could have used more checks and debug assertions from the beginning. It is also unfortunate that InnoDB originally did not initialize FIL_PAGE_TYPE on other than B-tree pages, and even then, it deferred the initialization until flushing the page to disk. Because of this, for uncompressed tables there is no check that a BLOB pointer is actually pointing to a BLOB page.

We have been hitting some BLOB-related bugs internally. One is that the BLOB pointer is pointing not to a first compressed BLOB page (ZBLOB) but to a subsequent compressed BLOB page (ZBLOB2). I have not been able to spend time on narrowing down this bug yet. One possibility is that the server is killed and restarted before btr_free_externally_stored_field() gets fully written to the redo log. Could you be hitting this one?

To resolve the problem at hand, can you give some more info, such as the complete 20-byte BLOB reference? Does BTR_EXTERN_LEN make sense? Is the transaction pointed to by DB_TRX_ID still active in trx_sys->trx_list?

No feedback was provided for this bug for over a month, so it is
being suspended automatically. If you are able to provide the
information that was originally requested, please do so and change
the status of the bug back to "Open".