Description:
When executing a RQG workload, Falcon asserted as follows:
#9 0x000000316a2c3dfa in __cxa_throw () from /usr/lib64/libstdc++.so.6
#10 0x0000000000a07dc7 in IO::readPage (this=0x7f3c8694ec90, bdb=0x7f3c867fa600) at IO.cpp:302
#11 0x00000000009bb899 in Cache::fetchPage (this=0x7f3c867bd400, dbb=0x7f3c8694ec88, pageNumber=1954, pageType=PAGE_btree, lockType=Shared) at Cache.cpp:289
#12 0x00000000009e1960 in Dbb::fetchPage (this=0x7f3c8694ec88, pageNumber=1954, pageType=PAGE_btree, lockType=Shared) at Dbb.cpp:219
#13 0x00000000009e1b88 in Dbb::handoffPage (this=0x7f3c8694ec88, bdb=0x7f3c8681c540, pageNumber=1954, pageType=PAGE_btree, lockType=Shared) at Dbb.cpp:268
#14 0x0000000000a0139f in IndexRootPage::findLeaf (dbb=0x7f3c8694ec88, indexId=36, rootPage=46, indexKey=0x15919a0, lockType=Shared, transId=0)
at IndexRootPage.cpp:248
#15 0x0000000000a0187e in IndexRootPage::scanIndex (dbb=0x7f3c8694ec88, indexId=36, rootPage=46, lowKey=0x15919a0, highKey=0x238b5b0, searchFlags=0,
transId=0, bitmap=0x7f3c865c57b0) at IndexRootPage.cpp:382
#16 0x00000000009fdabe in Index::scanIndex (this=0x7f3c865dd5c0, lowKey=0x0, highKey=0x238b5b0, searchFlags=0, transaction=0x2496a10, bitmap=0x7f3c865c57b0)
at Index.cpp:510
#17 0x000000000097e2ef in StorageDatabase::indexScan (this=0x7f3c86591208, index=0x7f3c865dd5c0, lower=0x0, upper=0x238b5a8, searchFlags=0,
storageConnection=0x7f3c866268b8, bitmap=0x7f3c865c57b0) at StorageDatabase.cpp:784
#18 0x00000000009865d0 in StorageTable::indexScan (this=0x2388af0, indexOrder=0) at StorageTable.cpp:259
#19 0x0000000000974ef3 in StorageInterface::scanRange (this=0x7f3c80f44580, start_key=0x0, end_key=0x7f3c80f27d70, eqRange=false) at ha_falcon.cpp:1805
#20 0x00000000009751fe in StorageInterface::read_range_first (this=0x7f3c80f44580, start_key=0x0, end_key=0x7f3c80f27d70, eq_range_arg=false, sorted=true)
at ha_falcon.cpp:1746
#21 0x00000000008268da in ha_partition::handle_ordered_index_scan (this=0x7f3c80f27c40, buf=0x7f3c80f82ca0 "ЪЪЪ", reverse_order=false)
at ha_partition.cc:4602
#22 0x000000000082702a in ha_partition::common_index_read (this=0x7f3c80f27c40, buf=0x7f3c80f82ca0 "ЪЪЪ", have_start_key=false) at ha_partition.cc:3979
#23 0x00000000008271d0 in ha_partition::read_range_first (this=0x7f3c80f27c40, start_key=0x0, end_key=0x7f3c80f27d40, eq_range_arg=false, sorted=true)
at ha_partition.cc:4235
#24 0x0000000000817fe4 in handler::multi_range_read_next (this=0x7f3c80f27c40, range_info=0x7f3c7230dc60) at handler.cc:4342
#25 0x00000000007f5a63 in QUICK_RANGE_SELECT::get_next (this=0x7f3c6c0aced0) at opt_range.cc:8558
#26 0x000000000081193a in rr_quick (info=0x7f3c6c0e2fd8) at records.cc:322
#27 0x000000000074d173 in join_init_read_record (tab=0x7f3c6c0e2f50) at sql_select.cc:17028
#28 0x00000000007509ab in sub_select (join=0x7f3c6c0d9ee0, join_tab=0x7f3c6c0e2f50, end_of_records=false) at sql_select.cc:16222
#29 0x000000000075e359 in do_select (join=0x7f3c6c0d9ee0, fields=0x0, table=0x7f3c6c0e4c70, procedure=0x0) at sql_select.cc:15786
#30 0x0000000000776ae7 in JOIN::exec (this=0x7f3c6c0d9ee0) at sql_select.cc:2463
#31 0x0000000000772ede in mysql_select (thd=0x22a1d88, rref_pointer_array=0x22a3d20, tables=0x228a8c0, wild_num=1, fields=@0x22a3c40, conds=0x228b0a8,
og_num=2, order=0x228b840, group=0x228b368, having=0x228b5a0, proc_param=0x0, select_options=2149337600, result=0x228b910, unit=0x22a36d0,
select_lex=0x22a3b38) at sql_select.cc:3062
#32 0x0000000000778687 in handle_select (thd=0x22a1d88, lex=0x22a3630, result=0x228b910, setup_tables_done_option=0) at sql_select.cc:314
#33 0x00000000006d5023 in execute_sqlcom_select (thd=0x22a1d88, all_tables=0x228a8c0) at sql_parse.cc:4768
#34 0x00000000006d6627 in mysql_execute_command (thd=0x22a1d88) at sql_parse.cc:2069
#35 0x00000000006de9ef in mysql_parse (thd=0x22a1d88,
inBuf=0x228a480 "SELECT * FROM `table10_falcon_key_pk_parts_2_int_autoinc` WHERE `char_64_latin1_not_null_key` <= 'like' GROUP BY `enum_utf8_not_null` HAVING `enum_latin1_not_null_key` <> 'tell' ORDER BY `int_unsigned"..., length=209, found_semicolon=0x7f3c7230ff00) at sql_parse.cc:5763
#36 0x00000000006df5da in dispatch_command (command=COM_QUERY, thd=0x22a1d88,
packet=0x7f3c6c00b019 "SELECT * FROM `table10_falcon_key_pk_parts_2_int_autoinc` WHERE `char_64_latin1_not_null_key` <= 'like' GROUP BY `enum_utf8_not_null` HAVING `enum_latin1_not_null_key` <> 'tell' ORDER BY `int_unsigned"..., packet_length=209) at sql_parse.cc:1009
#37 0x00000000006e0b03 in do_command (thd=0x22a1d88) at sql_parse.cc:691
#38 0x00000000006ce9cd in handle_one_connection (arg=0x22a1d88) at sql_connect.cc:1146
#39 0x000000315b0073da in start_thread () from /lib64/libpthread.so.0
#40 0x000000315a4e627d in clone () from /lib64/libc.so.6
The exception is:
299 throw SQLError(IO_ERROR,
300 "pread on file %s from page %d (offset %lld) returned %d bytes"
301 " instead of %d (possible read behind EOF)",
302 (const char*) fileName, bdb->pageNumber, (int64)offset, length, pageSize);
(gdb) print bdb->pageNumber
$20 = 1954
The offset is 7Mb beyond the end of file. The number 1954 comes from frame 14, IndexRootPage::findLeaf ,
(gdb) print node
$21 = {node = 0x7f3c8651a03c, nextNode = 0x7f3c8651a040, offset = 0, length = 0, key = 0x7f3c8651a040 "", number = 1954}
meaning that the index referenced a page that could not be read from disk.
How to repeat:
perl runall.pl --engine=Falcon --reporters=Deadlock,ErrorLog,Backtrace,Recovery,Shutdown --mysqld=--loose-falcon-lock-wait-timeout=1 --mysqld=--loose-innodb-lock-wait-timeout=1 --mysqld=--log-output=none --mysqld=--skip-safemalloc --mysqld=--transaction-isolation=READ-COMMITTED --mysqld=--falcon-page-size=32K --mem --rows=10 --threads=16 --mysqld=--falcon-scavenge-schedule='1 1 1 1 1' --basedir=/build/bzr/6.0-falcon-team --mask=59897 --queries=100000000 --duration=900 --gendata=conf/combinations.zz --grammar=conf/combinations.yy
Description: When executing a RQG workload, Falcon asserted as follows: #9 0x000000316a2c3dfa in __cxa_throw () from /usr/lib64/libstdc++.so.6 #10 0x0000000000a07dc7 in IO::readPage (this=0x7f3c8694ec90, bdb=0x7f3c867fa600) at IO.cpp:302 #11 0x00000000009bb899 in Cache::fetchPage (this=0x7f3c867bd400, dbb=0x7f3c8694ec88, pageNumber=1954, pageType=PAGE_btree, lockType=Shared) at Cache.cpp:289 #12 0x00000000009e1960 in Dbb::fetchPage (this=0x7f3c8694ec88, pageNumber=1954, pageType=PAGE_btree, lockType=Shared) at Dbb.cpp:219 #13 0x00000000009e1b88 in Dbb::handoffPage (this=0x7f3c8694ec88, bdb=0x7f3c8681c540, pageNumber=1954, pageType=PAGE_btree, lockType=Shared) at Dbb.cpp:268 #14 0x0000000000a0139f in IndexRootPage::findLeaf (dbb=0x7f3c8694ec88, indexId=36, rootPage=46, indexKey=0x15919a0, lockType=Shared, transId=0) at IndexRootPage.cpp:248 #15 0x0000000000a0187e in IndexRootPage::scanIndex (dbb=0x7f3c8694ec88, indexId=36, rootPage=46, lowKey=0x15919a0, highKey=0x238b5b0, searchFlags=0, transId=0, bitmap=0x7f3c865c57b0) at IndexRootPage.cpp:382 #16 0x00000000009fdabe in Index::scanIndex (this=0x7f3c865dd5c0, lowKey=0x0, highKey=0x238b5b0, searchFlags=0, transaction=0x2496a10, bitmap=0x7f3c865c57b0) at Index.cpp:510 #17 0x000000000097e2ef in StorageDatabase::indexScan (this=0x7f3c86591208, index=0x7f3c865dd5c0, lower=0x0, upper=0x238b5a8, searchFlags=0, storageConnection=0x7f3c866268b8, bitmap=0x7f3c865c57b0) at StorageDatabase.cpp:784 #18 0x00000000009865d0 in StorageTable::indexScan (this=0x2388af0, indexOrder=0) at StorageTable.cpp:259 #19 0x0000000000974ef3 in StorageInterface::scanRange (this=0x7f3c80f44580, start_key=0x0, end_key=0x7f3c80f27d70, eqRange=false) at ha_falcon.cpp:1805 #20 0x00000000009751fe in StorageInterface::read_range_first (this=0x7f3c80f44580, start_key=0x0, end_key=0x7f3c80f27d70, eq_range_arg=false, sorted=true) at ha_falcon.cpp:1746 #21 0x00000000008268da in ha_partition::handle_ordered_index_scan (this=0x7f3c80f27c40, buf=0x7f3c80f82ca0 "ЪЪЪ", reverse_order=false) at ha_partition.cc:4602 #22 0x000000000082702a in ha_partition::common_index_read (this=0x7f3c80f27c40, buf=0x7f3c80f82ca0 "ЪЪЪ", have_start_key=false) at ha_partition.cc:3979 #23 0x00000000008271d0 in ha_partition::read_range_first (this=0x7f3c80f27c40, start_key=0x0, end_key=0x7f3c80f27d40, eq_range_arg=false, sorted=true) at ha_partition.cc:4235 #24 0x0000000000817fe4 in handler::multi_range_read_next (this=0x7f3c80f27c40, range_info=0x7f3c7230dc60) at handler.cc:4342 #25 0x00000000007f5a63 in QUICK_RANGE_SELECT::get_next (this=0x7f3c6c0aced0) at opt_range.cc:8558 #26 0x000000000081193a in rr_quick (info=0x7f3c6c0e2fd8) at records.cc:322 #27 0x000000000074d173 in join_init_read_record (tab=0x7f3c6c0e2f50) at sql_select.cc:17028 #28 0x00000000007509ab in sub_select (join=0x7f3c6c0d9ee0, join_tab=0x7f3c6c0e2f50, end_of_records=false) at sql_select.cc:16222 #29 0x000000000075e359 in do_select (join=0x7f3c6c0d9ee0, fields=0x0, table=0x7f3c6c0e4c70, procedure=0x0) at sql_select.cc:15786 #30 0x0000000000776ae7 in JOIN::exec (this=0x7f3c6c0d9ee0) at sql_select.cc:2463 #31 0x0000000000772ede in mysql_select (thd=0x22a1d88, rref_pointer_array=0x22a3d20, tables=0x228a8c0, wild_num=1, fields=@0x22a3c40, conds=0x228b0a8, og_num=2, order=0x228b840, group=0x228b368, having=0x228b5a0, proc_param=0x0, select_options=2149337600, result=0x228b910, unit=0x22a36d0, select_lex=0x22a3b38) at sql_select.cc:3062 #32 0x0000000000778687 in handle_select (thd=0x22a1d88, lex=0x22a3630, result=0x228b910, setup_tables_done_option=0) at sql_select.cc:314 #33 0x00000000006d5023 in execute_sqlcom_select (thd=0x22a1d88, all_tables=0x228a8c0) at sql_parse.cc:4768 #34 0x00000000006d6627 in mysql_execute_command (thd=0x22a1d88) at sql_parse.cc:2069 #35 0x00000000006de9ef in mysql_parse (thd=0x22a1d88, inBuf=0x228a480 "SELECT * FROM `table10_falcon_key_pk_parts_2_int_autoinc` WHERE `char_64_latin1_not_null_key` <= 'like' GROUP BY `enum_utf8_not_null` HAVING `enum_latin1_not_null_key` <> 'tell' ORDER BY `int_unsigned"..., length=209, found_semicolon=0x7f3c7230ff00) at sql_parse.cc:5763 #36 0x00000000006df5da in dispatch_command (command=COM_QUERY, thd=0x22a1d88, packet=0x7f3c6c00b019 "SELECT * FROM `table10_falcon_key_pk_parts_2_int_autoinc` WHERE `char_64_latin1_not_null_key` <= 'like' GROUP BY `enum_utf8_not_null` HAVING `enum_latin1_not_null_key` <> 'tell' ORDER BY `int_unsigned"..., packet_length=209) at sql_parse.cc:1009 #37 0x00000000006e0b03 in do_command (thd=0x22a1d88) at sql_parse.cc:691 #38 0x00000000006ce9cd in handle_one_connection (arg=0x22a1d88) at sql_connect.cc:1146 #39 0x000000315b0073da in start_thread () from /lib64/libpthread.so.0 #40 0x000000315a4e627d in clone () from /lib64/libc.so.6 The exception is: 299 throw SQLError(IO_ERROR, 300 "pread on file %s from page %d (offset %lld) returned %d bytes" 301 " instead of %d (possible read behind EOF)", 302 (const char*) fileName, bdb->pageNumber, (int64)offset, length, pageSize); (gdb) print bdb->pageNumber $20 = 1954 The offset is 7Mb beyond the end of file. The number 1954 comes from frame 14, IndexRootPage::findLeaf , (gdb) print node $21 = {node = 0x7f3c8651a03c, nextNode = 0x7f3c8651a040, offset = 0, length = 0, key = 0x7f3c8651a040 "", number = 1954} meaning that the index referenced a page that could not be read from disk. How to repeat: perl runall.pl --engine=Falcon --reporters=Deadlock,ErrorLog,Backtrace,Recovery,Shutdown --mysqld=--loose-falcon-lock-wait-timeout=1 --mysqld=--loose-innodb-lock-wait-timeout=1 --mysqld=--log-output=none --mysqld=--skip-safemalloc --mysqld=--transaction-isolation=READ-COMMITTED --mysqld=--falcon-page-size=32K --mem --rows=10 --threads=16 --mysqld=--falcon-scavenge-schedule='1 1 1 1 1' --basedir=/build/bzr/6.0-falcon-team --mask=59897 --queries=100000000 --duration=900 --gendata=conf/combinations.zz --grammar=conf/combinations.yy