MySQL Bugs: #55782: ut_ad assert page_zip_get_size(&block->page.zip) == zip_size buf_page_get

Bug #55782	ut_ad assert page_zip_get_size(&block->page.zip) == zip_size buf_page_get_gen()
Submitted:	5 Aug 2010 14:48	Modified:	10 Jan 2011 22:07
Reporter:	Mikhail Izioumtchenko	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S3 (Non-critical)
Version:	5.6	OS:	Any
Assigned to:	Inaam Rana	CPU Architecture:	Any

Description:
Private bug because this is a ut_ad() assert that requires UNIV_DEBUG

in a stress test:

InnoDB: Assertion failure in thread 1224685888 in file /spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/buf/buf0buf.c line 2989
InnoDB: Failing assertion: page_zip_get_size(&block->page.zip) == zip_size

 #5  0x0000003acbc31d10 in abort () from /lib64/libc.so.6
 #6  0x00000000008d415d in buf_page_get_gen (space=3, zip_size=8192, offset=23,
     rw_latch=2, guess=0x0, mode=11,
     file=0xb4a250 "/spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/row/row0ins.c", line=2002, mtr=0x48ff0780)
     at /spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/buf/buf0buf.c:2989
 #7  0x00000000008bc7ca in btr_cur_search_to_nth_level (index=0x181ba118,
     level=0, tuple=0x1826bde8, mode=4, latch_mode=2, cursor=0x48ff0c60,
     has_search_latch=0,
     file=0xb4a250 "/spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/row/row0ins.c", line=2002, mtr=0x48ff0780)
     at /spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/btr/btr0cur.c:574
 #8  0x0000000000968b19 in row_ins_index_entry_low (mode=2, index=0x181ba118,
     entry=0x1826bde8, n_ext=0, thr=0x180ec338)
     at /spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/row/row0ins.c:2000
 #9  0x0000000000969145 in row_ins_index_entry (index=0x181ba118,
     entry=0x1826bde8, n_ext=0, foreign=1, thr=0x180ec338)
     at /spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/row/row0ins.c:2171
 #10 0x0000000000969446 in row_ins_index_entry_step (node=0x2aaab819be30,
     thr=0x180ec338)
     at /spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/row/row0ins.c:2256
 #11 0x00000000009696c6 in row_ins (node=0x2aaab819be30, thr=0x180ec338)
     at /spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/row/row0ins.c:2388
 #12 0x0000000000969966 in row_ins_step (thr=0x180ec338)
     at /spare2/mizioumt/ctc/mysql_src_c56/storage/innobase/row/row0ins.c:2498
 #13 0x0000000000874f01 in row_insert_for_mysql (

How to repeat:
InnoDB's internal stress/recovery test. Reproducibility is fairly low,
reproduces in mysql-next-mr-innodb only. 

Suggested fix:
I believe the assert is not good under certain circumstances, see note

The failing assertion

ut_ad(page_zip_get_size(&block->page.zip) == zip_size);

should be moved after mutex_enter(block_mutex) or even mutex_exit(hash_mutex). Otherwise it will be performing a dirty read.

After discussing this with Inaam and reviewing the code, I have an explanation for the assertion failure. page_zip_get_size() can only be changed by page_zip_set_size(). I reviewed all page_zip_set_size() calls in the code.

When the function buf_page_init_for_read() allocates a block and a compressed page frame, it does that in two steps. First, it allocates the uncompressed block and adds it to page_hash. Then, while still holding hash_mutex, it acquires block->mutex and releases hash_mutex. Some time after that, it calls page_zip_set_size(). My initial suggestion was correct: the offending assertion was invoking page_zip_get_size() too early, before acquiring block_mutex. The assertion holds in the core dump, because the thread in buf_page_init_for_read() finally got around to invoking page_zip_set_size().

Other occurrences of page_zip_set_size() look fine: the zip.ssize is not being modified while the page resides in the buf_pool->page_hash.

The question arises whether there are further problems. After buf_page_init_for_read() calls page_zip_set_size(), it releases the block->mutex. But, it did also I/O-fix the block for reading (BUF_IO_READ), telling buf_page_get_gen() to steer clear of it. Then it allocates the compressed page frame and acquires block->mutex shortly to make zip.data point to it.

Pushed into mysql-trunk 5.6.99-m5 (revid:alexander.nozdrin@oracle.com-20101113155825-czmva9kg4n31anmu) (version source revid:alexander.nozdrin@oracle.com-20101113152450-2zzcm50e7i4j35v7) (merge vers: 5.6.1-m4) (pib:21)

Pushed into mysql-next-mr (revid:alexander.nozdrin@oracle.com-20101113160336-atmtmfb3mzm4pz4i) (version source revid:vasil.dimov@oracle.com-20100629074804-359l9m9gniauxr94) (pib:21)

Pushed into mysql-trunk 5.6.1 (revid:alexander.nozdrin@oracle.com-20101205122447-6x94l4fmslpbttxj) (version source revid:alexander.nozdrin@oracle.com-20101205122447-6x94l4fmslpbttxj) (merge vers: 5.6.1) (pib:23)