MySQL Bugs: #75124: Failing assertion: page_is_comp(get_block->frame) == page_is

Bug #75124	Failing assertion: page_is_comp(get_block->frame) == page_is_comp(page)
Submitted:	6 Dec 2014 3:06	Modified:	7 Dec 2014 7:34
Reporter:	zhai weixiang (OCA)	Email Updates:
Status:	Duplicate	Impact on me:	None
Category:	MySQL Server: InnoDB storage engine	Severity:	S3 (Non-critical)
Version:	5.7.5	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
While testing 5.7.5, I got an assert failure as follows:

2014-12-06 10:52:50 0x2b8a10659700  InnoDB: Assertion failure in thread 47871980574464 in file btr0cur.cc line 307
InnoDB: Failing assertion: page_is_comp(get_block->frame) == page_is_comp(page)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
02:52:50 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=131
max_threads=21000
thread_count=128
connection_count=128
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8334199 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x2b897d829000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 2b8a10658e08 thread_stack 0x80000
/u01/my575/bin/mysqld(my_print_stacktrace+0x35)[0xabbbe5]
/u01/my575/bin/mysqld(handle_fatal_signal+0x43b)[0x61edcb]
/lib64/libpthread.so.0[0x372ca0f500]
/lib64/libc.so.6(gsignal+0x35)[0x372c6328a5]
/lib64/libc.so.6(abort+0x175)[0x372c634085]
/u01/my575/bin/mysqld(_Z18ut_print_timestampP8_IO_FILE+0x0)[0x60f271]
/u01/my575/bin/mysqld[0xcbae2e]
/u01/my575/bin/mysqld(_Z27btr_cur_search_to_nth_levelP12dict_index_tmPK8dtuple_tmmP9btr_cur_tmPKcmP5mtr_t+0x1bac)[0xcbf02c]
/u01/my575/bin/mysqld(_Z29row_ins_clust_index_entry_lowmmP12dict_index_tmP8dtuple_tmP9que_thr_tb+0x255)[0xc14c65]
/u01/my575/bin/mysqld(_Z12row_ins_stepP9que_thr_t+0x77e)[0xc1aa9e]
/u01/my575/bin/mysqld(_Z20row_insert_for_mysqlPKhP14row_prebuilt_t+0x389)[0xc29609]
/u01/my575/bin/mysqld(_ZN11ha_innobase9write_rowEPh+0x242)[0xb908a2]
/u01/my575/bin/mysqld(_ZN7handler12ha_write_rowEPh+0x104)[0x665754]
/u01/my575/bin/mysqld(_Z12write_recordP3THDP5TABLEP9COPY_INFOS4_+0x71)[0xa0f5e1]
2014-12-06T02:52:50.447639Z 3089 [Note] Access denied for user 'root'@'127.0.0.1' (using password: NO)
/u01/my575/bin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicates+0x888)[0xa12098]
/u01/my575/bin/mysqld(_Z21mysql_execute_commandP3THD+0x19e5)[0x8b7855]
/u01/my575/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x4a0)[0x8bb4b0]
/u01/my575/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcm+0x148e)[0x8bcade]
/u01/my575/bin/mysqld(handle_connection+0x254)[0x978c74]
/u01/my575/bin/mysqld(pfs_spawn_thread+0x127)[0xacd9d7]
/lib64/libpthread.so.0[0x372ca07851]
/lib64/libc.so.6(clone+0x6d)[0x372c6e767d]

How to repeat:
The table used to test:

CREATE TABLE `t1` (
  `a` int(11) NOT NULL AUTO_INCREMENT,
  `b` int(11) DEFAULT NULL,
  `c` int(11) DEFAULT NULL,
  `d` varchar(256) DEFAULT NULL,
  `e` blob,
  PRIMARY KEY (`a`),
  KEY `b` (`b`,`d`(60)),
  KEY `c` (`c`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED;

Create 10 table (t1~t10) with same schema.

The lua:

$cat sysbench/tests/db/oltp_simple_ins3.lua
pathtest = string.match(test, "(.*/)") or ""

dofile(pathtest .. "common.lua")

function thread_init(thread_id)
   set_vars()

end

function event(thread_id)
   local table_name
   table_name = "t".. sb_rand_uniform(1, 10)

   local str1 = sb_rand_str("###########-###########-###########-###########-###########-###########-###########-###########-###########-###########")
   local str2 = sb_rand_str([[###########-###########-###########-###########-###########]])
   db_query("INSERT INTO "..table_name.."(b,c,d,e) VALUES(rand()*10000,rand()*1000000,'"..str1.."',repeat('"..str2.."',100))");
end

Then use sysbench with this specified lua:

sysbench/sysbench --debug=off --test=sysbench/tests/db/oltp_simple_ins3.lua --oltp-tables-count=100  --oltp-point-selects=0 --oltp-table-size=100000 --num-threads=$1 --max-requests=10000000000 --max-time=7200000 --oltp-auto-inc=off --mysql-engine-trx=yes --mysql-table-engine=innodb --oltp-test-mod=complex --mysql-db=$2  --mysql-host=$host  --mysql-port=$port --mysql-user=$user_name --oltp-user-delay-min=10 --oltp-user-delay-max=100 run

I can repeat the crash after about one hour running the tests.

Suggested fix:
I don't know

I got another crash while running the tests on 5.6.16(slightly patched):

014-12-06 02:16:43 2adb6980c700  InnoDB: Assertion failure in thread 47121856251648 in file buf0flu.cc line 935
InnoDB: Failing assertion: page_zip_verify_checksum(frame, zip_size)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
18:16:43 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=132
max_threads=21001
thread_count=129
connection_count=129
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 492478672 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x80000
/u01/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x8f9a15]
/u01/mysql/bin/mysqld(handle_fatal_signal+0x41b)[0x659deb]
/lib64/libpthread.so.0[0x372ca0f500]
/lib64/libc.so.6(gsignal+0x35)[0x372c6328a5]
/lib64/libc.so.6(abort+0x175)[0x372c634085]
/u01/mysql/bin/mysqld[0xab044c]
/u01/mysql/bin/mysqld[0xab1fa5]
/u01/mysql/bin/mysqld[0xab261a]
/u01/mysql/bin/mysqld[0xab2e0b]
/u01/mysql/bin/mysqld[0xab38d1]
/lib64/libpthread.so.0[0x372ca07851]
/lib64/libc.so.6(clone+0x6d)[0x372c6e767d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

And backtrace that raised the assert:
#6  0x0000000000659ce7 in handle_fatal_signal (sig=6) at /u01/project/tmp/rds_5616/sql/signal_handler.cc:210
#7  <signal handler called>
#8  0x000000372c6328a5 in raise () from /lib64/libc.so.6
#9  0x000000372c634085 in abort () from /lib64/libc.so.6
#10 0x0000000000ab044c in buf_flush_write_block_low (buf_pool=<value optimized out>, bpage=0x2adb706363a0, flush_type=BUF_FLUSH_LIST, sync=false) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:935
#11 buf_flush_page (buf_pool=<value optimized out>, bpage=0x2adb706363a0, flush_type=BUF_FLUSH_LIST, sync=false) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:1072
#12 0x0000000000ab1fa5 in buf_flush_try_neighbors (bpage=<value optimized out>, flush_type=BUF_FLUSH_LIST, n_to_flush=750, count=0x2adb6980bc98) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:1275
#13 buf_flush_page_and_try_neighbors (bpage=<value optimized out>, flush_type=BUF_FLUSH_LIST, n_to_flush=750, count=0x2adb6980bc98) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:1352
#14 0x0000000000ab261a in buf_do_flush_list_batch (buf_pool=0x2ad2a73fe9c8, flush_type=<value optimized out>, min_n=750, lsn_limit=3881870909195) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:1627
#15 buf_flush_batch (buf_pool=0x2ad2a73fe9c8, flush_type=<value optimized out>, min_n=750, lsn_limit=3881870909195) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:1702
#16 0x0000000000ab2e0b in buf_flush_list (min_n=750, lsn_limit=3881870909195, n_processed=0x2adb6980bdf0) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:1939
#17 0x0000000000ab38d1 in page_cleaner_do_flush_batch (arg=<value optimized out>) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:2168
#18 page_cleaner_flush_pages_if_needed (arg=<value optimized out>) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:2332
#19 buf_flush_page_cleaner_thread (arg=<value optimized out>) at /u01/project/tmp/rds_5616/storage/innobase/buf/buf0flu.cc:2426
#20 0x000000372ca07851 in start_thread () from /lib64/libpthread.so.0
#21 0x000000372c6e767d in clone () from /lib64/libc.so.6
(gdb) 

And the server event can't recover from the crash.

Assert backtrace (5.7.5):

#0  0x000000372c6328a5 in raise () from /lib64/libc.so.6
#1  0x000000372c634085 in abort () from /lib64/libc.so.6
#2  0x000000000060f271 in ut_dbg_assertion_failed (expr=0xfbf998 "page_is_comp(get_block->frame) == page_is_comp(page)", file=<value optimized out>, line=307)
    at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/ut/ut0dbg.cc:68
#3  0x0000000000cbae2e in btr_cur_latch_leaves (block=<value optimized out>, page_id=..., page_size=<value optimized out>, latch_mode=<value optimized out>, cursor=0x2b1eec500f90, mtr=0x2b1eec500490)
    at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/btr/btr0cur.cc:306
#4  0x0000000000cbf02c in btr_cur_search_to_nth_level (index=0x2b1ecfa60500, level=0, tuple=0x2b1e5d970a80, mode=4, latch_mode=33, cursor=0x2b1eec500f90, has_search_latch=0,
    file=0xfad090 "/u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/row/row0ins.cc", line=2372, mtr=0x2b1eec500490) at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/btr/btr0cur.cc:1288
#5  0x0000000000c14c65 in btr_pcur_open_low (flags=0, mode=33, index=0x2b1ecfa60500, n_uniq=1, entry=0x2b1e5d970a80, n_ext=0, thr=0x2b1e5d8e7bd8, dup_chk_only=false)
    at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/include/btr0pcur.ic:455
#6  row_ins_clust_index_entry_low (flags=0, mode=33, index=0x2b1ecfa60500, n_uniq=1, entry=0x2b1e5d970a80, n_ext=0, thr=0x2b1e5d8e7bd8, dup_chk_only=false)
    at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/row/row0ins.cc:2372
#7  0x0000000000c1aa9e in row_ins_clust_index_entry (thr=0x2b1e5d8e7bd8) at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/row/row0ins.cc:3208
#8  row_ins_index_entry (thr=0x2b1e5d8e7bd8) at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/row/row0ins.cc:3306
#9  row_ins_index_entry_step (thr=0x2b1e5d8e7bd8) at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/row/row0ins.cc:3448
#10 row_ins (thr=0x2b1e5d8e7bd8) at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/row/row0ins.cc:3590
#11 row_ins_step (thr=0x2b1e5d8e7bd8) at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/row/row0ins.cc:3762
#12 0x0000000000c29609 in row_insert_for_mysql_using_ins_graph (mysql_rec=<value optimized out>, prebuilt=<value optimized out>) at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/row/row0mysql.cc:1579
#13 row_insert_for_mysql (mysql_rec=<value optimized out>, prebuilt=<value optimized out>) at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/row/row0mysql.cc:1695
#14 0x0000000000b908a2 in ha_innobase::write_row (this=0x2b1e5d862420, record=0x2b1e5d877320 "�\016�)") at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/handler/ha_innodb.cc:6755
#15 0x0000000000665754 in handler::ha_write_row (this=0x2b1e5d862420, buf=0x2b1e5d877320 "�\016�)") at /u01/project/mysql-lab/mysql-5.7.5-m15/sql/handler.cc:7423
#16 0x0000000000a0f5e1 in write_record (thd=0x2b1e5d8fb000, table=0x2b1e5d841810, info=0x2b1eec501550, update=0x2b1eec5014d0) at /u01/project/mysql-lab/mysql-5.7.5-m15/sql/sql_insert.cc:1679
#17 0x0000000000a12098 in mysql_insert (thd=0x2b1e5d8fb000, table_list=0x2b1e5d906b10, fields=..., values_list=..., update_fields=<value optimized out>, update_values=<value optimized out>, duplic=DUP_ERROR)
    at /u01/project/mysql-lab/mysql-5.7.5-m15/sql/sql_insert.cc:707
#18 0x00000000008b7855 in mysql_execute_command (thd=0x2b1e5d8fb000) at /u01/project/mysql-lab/mysql-5.7.5-m15/sql/sql_parse.cc:3367
#19 0x00000000008bb4b0 in mysql_parse (thd=0x2b1e5d8fb000, parser_state=<value optimized out>) at /u01/project/mysql-lab/mysql-5.7.5-m15/sql/sql_parse.cc:5427
#20 0x00000000008bcade in dispatch_command (command=COM_QUERY, thd=0x2b1e5d8fb000, packet=0x2b1eec502ac0 "", packet_length=255) at /u01/project/mysql-lab/mysql-5.7.5-m15/sql/sql_parse.cc:1250
#21 0x0000000000978c74 in handle_connection (arg=<value optimized out>) at /u01/project/mysql-lab/mysql-5.7.5-m15/sql/conn_handler/connection_handler_per_thread.cc:298
#22 0x0000000000acd9d7 in pfs_spawn_thread (arg=0x2b1e53dfa2d0) at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/perfschema/pfs.cc:2137
#23 0x000000372ca07851 in start_thread () from /lib64/libpthread.so.0
#24 0x000000372c6e767d in clone () from /lib64/libc.so.6
(gdb) f 3
#3  0x0000000000cbae2e in btr_cur_latch_leaves (block=<value optimized out>, page_id=..., page_size=<value optimized out>, latch_mode=<value optimized out>, cursor=0x2b1eec500f90, mtr=0x2b1eec500490)
    at /u01/project/mysql-lab/mysql-5.7.5-m15/storage/innobase/btr/btr0cur.cc:306
306                             ut_a(page_is_comp(get_block->frame)
(gdb) p block
$3 = <value optimized out>
(gdb) p get_block
$4 = (buf_block_t *) 0x0

Thank you for the report.
Imho this is duplicate of Bug #74596, please see Bug #74596

Thanks,
Umesh

Hi, Umesh,  It's indeed a duplicate of bug#74596 ! Thanks ! 

I'll verify another assert (Failing assertion: page_zip_verify_checksum(frame, zip_size) )  with the latest version of 5.6  and then update this bug report.

may be related to http://bugs.mysql.com/bug.php?id=74596

Thank you Zhai for confirming.
Marking this as duplicate of Bug #74596

Thanks,
Umesh