| Bug #58404 | Assertion prev_view == NULL || prev_view->low_limit_no >= view->low_limit_no | ||
|---|---|---|---|
| Submitted: | 22 Nov 2010 21:46 | Modified: | 15 Oct 2012 13:30 |
| Reporter: | Elena Stepanova | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: InnoDB storage engine | Severity: | S2 (Serious) |
| Version: | 5.6.1-m5-debug | OS: | Any |
| Assigned to: | Sunny Bains | CPU Architecture: | Any |
[24 Nov 2010 22:47]
Elena Stepanova
The problem is reproducible on debug builds with long high-concurrency sysbench workload. Every test run that we have crashes at some point, although sometimes it happens after ~20 min of the test, while sometimes it might be several hours. I did not get the failure on a non-debug version (1 test run, 8 hours). How to reproduce: Start server (only system tables in the data folder): mysqld --no-defaults --basedir=<basedir> --datadir=<data folder> --log-error=mysqld.err --innodb_buffer_pool_size=256M --max_connections=3000 --max_prepared_stmt_count=21100 Run sysbench prepare (I used sysbench 0.4.12): sysbench --test=oltp --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test --mysql-table-engine=innodb --oltp-table-size=100000 prepare Run sysbench workflow: sysbench --test=oltp --num-threads=2000 --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=test --mysql-table-engine=innodb --max-requests=0 --max-time=28800 run
[1 Dec 2010 3:06]
Sunny Bains
When creating the read view we set view->low_limit_no to max(trx_sys->trx_list : no) and
transactions can change state from RUNNING to COMMITED while we are iterating over the
trx_sys_t::trx_list. Therefore it is possible for concurrently running threads to then end up
inserting the new view out of order in the trx_sys_t::view_list.
328 for (trx = UT_LIST_GET_FIRST(trx_sys->trx_list);
329 trx != NULL;
330 trx = UT_LIST_GET_NEXT(trx_list, trx)) {
331
332 /* Note: We are doing a dirty read of the trx_t::state
333 without the cover of the trx_t::mutex. The state change
334 to TRX_STATE_PREPARED is done using only the trx_t::mutex. * /
335
336 if (trx->id != cr_trx_id
337 && (trx->state == TRX_STATE_ACTIVE
338 || trx->state == TRX_STATE_PREPARED)) {
339
340 ut_ad(n_trx < view->n_trx_ids);
341
342 view->trx_ids[n_trx++] = trx->id;
343
344 /* NOTE that a transaction whose trx number is <
345 trx_sys->max_trx_id can still be active, if it is
346 in the middle of its commit! Note that when a
347 transaction starts, we initialize trx->no to
348 IB_ULONGLONG_MAX. */
349
350 if (view->low_limit_no > trx->no) {
351
352 view->low_limit_no = trx->no;
353 }
354 }
355 }
356
357 view->n_trx_ids = n_trx;
358
359 if (view->n_trx_ids > 0) {
360 /* The last active transaction has the smallest id: */
361
362 view->up_limit_id = view->trx_ids[view->n_trx_ids - 1];
363 } else {
364 view->up_limit_id = view->low_limit_id;
365 }
366
367 ut_ad(read_view_validate(view));
368
369 mutex_enter(&trx_sys->read_view_mutex);
370
371 UT_LIST_ADD_FIRST(view_list, trx_sys->view_list, view);
372
373 mutex_exit(&trx_sys->read_view_mutex);
By the time a thread gets to line 371 there is no guarantee that when it prepends
the view to trx_sys->view list it will preserve the order. There are several solutions,
in order of decreasing preference:
1. Ordered insert based on view->low_limit_no
2. Force thread to acquire trx_sys_t::lock in X mode in lock_trx_release_locks()
when changing trx_t::state to TRX_STATE_COMMITTED_IN_MEMORY
3. Hold the trx_sys_t::read_view_mutex for the duration of the for loop
The reason I prefer 1 is because the worst case should require the traversal of as many
list nodes as there are threads doing concurrent create of views. This number should be small.
2. Is also OK, it is simple but my worry is increasing the contention on the trx_sys_t::lock. For 3
my concern is that with > 1024 threads it could make read view create a bottleneck.
[22 Dec 2010 21:31]
Bugs System
Pushed into mysql-trunk 5.6.1 (revid:alexander.nozdrin@oracle.com-20101222212842-y0t3ibtd32wd9qaw) (version source revid:alexander.nozdrin@oracle.com-20101222212842-y0t3ibtd32wd9qaw) (merge vers: 5.6.1) (pib:24)
Description: InnoDB: Assertion failure in thread 1228626240 in file mysql-5.6.1-m5/storage/innobase/read/read0read.c line 219 InnoDB: Failing assertion: prev_view == NULL || prev_view->low_limit_no >= view->low_limit_no #5 0x00002b847d990fb0 in abort () from /lib64/libc.so.6 #6 0x0000000000974c92 in read_view_list_validate () at mysql-5.6.1-m5/storage/innobase/read/read0read.c:218 #7 0x0000000000975274 in read_view_open_now_low (cr_trx_id=1030411, heap=0xa903970) at mysql-5.6.1-m5/storage/innobase/read/read0read.c:375 #8 0x00000000009752e5 in read_view_open_now (cr_trx_id=1030411, heap=0xa903970) at mysql-5.6.1-m5/storage/innobase/read/read0read.c:397 #9 0x00000000009b5a9f in trx_assign_read_view (trx=0xa25fb28) at mysql-5.6.1-m5/storage/innobase/trx/trx0trx.c:911 #10 0x000000000099104f in row_search_for_mysql (buf=0x2aaac5160510 "Ъ", mode=2, prebuilt=0x2aaabffd3b58, match_mode=1, direction=0) at mysql-5.6.1-m5/storage/innobase/row/row0sel.c:3921 #11 0x000000000096e0a9 in ha_innobase::index_read (this=0x2aaabee62190, buf=0x2aaac5160510 "Ъ", key_ptr=0x2f5fd28 "╙\023", key_len=4, find_flag=HA_READ_KEY_EXACT) at mysql-5.6.1-m5/storage/innobase/handler/ha_innodb.cc:6078 #12 0x000000000073d1da in handler::index_read_map (this=0x2aaabee62190, buf=0x2aaac5160510 "Ъ", key=0x2f5fd28 "╙\023", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at mysql-5.6.1-m5/sql/handler.h:1806 #13 0x00000000007315fd in handler::index_read_idx_map (this=0x2aaabee62190, buf=0x2aaac5160510 "Ъ", index=0, key=0x2f5fd28 "╙\023", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at mysql-5.6.1-m5/sql/handler.cc:5425 #14 0x000000000073256c in handler::ha_index_read_idx_map (this=0x2aaabee62190, buf=0x2aaac5160510 "Ъ", index=0, key=0x2f5fd28 "╙\023", keypart_map=1, find_flag=HA_READ_KEY_EXACT) at mysql-5.6.1-m5/sql/handler.cc:2261 #15 0x000000000061103d in join_read_const (tab=0x2f5f4e0) at mysql-5.6.1-m5/sql/sql_select.cc:17879 #16 0x000000000061810a in join_read_const_table (tab=0x2f5f4e0, pos=0x2f5f990) at mysql-5.6.1-m5/sql/sql_select.cc:17774 #17 0x000000000061cf9d in make_join_statistics (join=0x2f5ee38, tables_arg=0x129965b0, conds=0x2f5f2c8, keyuse_array=0x2f5f0a8) at mysql-5.6.1-m5/sql/sql_select.cc:4816 #18 0x0000000000620b71 in JOIN::optimize (this=0x2f5ee38) at mysql-5.6.1-m5/sql/sql_select.cc:1951 #19 0x0000000000624f6b in mysql_select (thd=0x2f4b040, rref_pointer_array=0x12995b00, tables=0x129965b0, wild_num=0, fields=@0x12995a20, conds=0x12997250, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2417232640, result=0x129970f0, unit=0x129952e0, select_lex=0x12995918) at mysql-5.6.1-m5/sql/sql_select.cc:3484 #20 0x000000000062ad8b in handle_select (thd=0x2f4b040, lex=0x12995230, result=0x129970f0, setup_tables_done_option=0) at mysql-5.6.1-m5/sql/sql_select.cc:323 #21 0x00000000005cc6b2 in execute_sqlcom_select (thd=0x2f4b040, all_tables=0x129965b0) at mysql-5.6.1-m5/sql/sql_parse.cc:4477 #22 0x00000000005cd61b in mysql_execute_command (thd=0x2f4b040) at mysql-5.6.1-m5/sql/sql_parse.cc:2073 #23 0x00000000005e9f4d in Prepared_statement::execute (this=0x1214e650, expanded_query=0x493b42e0, open_cursor=false) at mysql-5.6.1-m5/sql/sql_prepare.cc:3778 #24 0x00000000005ea2a1 in Prepared_statement::execute_loop (this=0x1214e650, expanded_query=0x493b42e0, open_cursor=false, packet=0x2f4eb2a "", packet_end=0x2f4eb30 "") at mysql-5.6.1-m5/sql/sql_prepare.cc:3431 #25 0x00000000005ea754 in mysqld_stmt_execute (thd=0x2f4b040, packet_arg=0x2f4eb21 "\001", packet_length=15) at mysql-5.6.1-m5/sql/sql_prepare.cc:2574 #26 0x00000000005d554c in dispatch_command (command=COM_STMT_EXECUTE, thd=0x2f4b040, packet=0x2f4eb21 "\001", packet_length=15) at mysql-5.6.1-m5/sql/sql_parse.cc:1011 #27 0x00000000005d6c01 in do_command (thd=0x2f4b040) at mysql-5.6.1-m5/sql/sql_parse.cc:799 #28 0x00000000006a628b in do_handle_one_connection (thd_arg=0x2f4b040) at mysql-5.6.1-m5/sql/sql_connect.cc:745 #29 0x00000000006a6387 in handle_one_connection (arg=0x2f4b040) at mysql-5.6.1-m5/sql/sql_connect.cc:684 #30 0x000000000095aebe in pfs_spawn_thread (arg=0x2f52b30) at mysql-5.6.1-m5/storage/perfschema/pfs.cc:1078 #31 0x00002b847d2af143 in start_thread () from /lib64/libpthread.so.0 #32 0x00002b847da208cd in clone () from /lib64/libc.so.6 #33 0x0000000000000000 in ?? () thd->query at 0x2f5ed30 = SELECT c from sbtest where id=5034 thd->thread_id=239 thd->killed=NOT_KILLED How to repeat: The failure was observed in sysbench-based high-concurrency test. If it is reproducible, the test case will be provided