| Bug #76650 | sysbench read/write crashes on POWER | ||
|---|---|---|---|
| Submitted: | 10 Apr 2015 8:21 | Modified: | 5 Nov 2018 13:54 |
| Reporter: | Stewart Smith | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: InnoDB storage engine | Severity: | S1 (Critical) |
| Version: | 5.7.7 | OS: | Linux (Ubuntu Vivid ppc64el) |
| Assigned to: | CPU Architecture: | Any | |
| Tags: | power, PowerPC | ||
[8 Nov 2017 5:31]
Daniel Black
On 8286-42A Power8 in SMT8
tested 5.7.20 running:
bin/mysqld --no-defaults --pid-file=/tmp/master.pid --port 3310 --datadir=/dev/shm/oltp_master --socket /tmp/oltp_master.sock --log-bin=/dev/shm/oltp_master/mysqlbin --server-id=120 --innodb_log_file_size=200M --innodb_log_files_in_group=5 --innodb-buffer-pool-size=6G --innodb-buffer-pool-instances=6 --innodb_file_per_table --binlog_checksum=NONE --innodb_checksum_algorithm=none --max-connections=16000 --max_prepared_stmt_count=1M --core-file
With sysbench as follows:
export try=50
sock=/tmp/oltp_master.sock
engine=innodb
export sysbench_options="--db-driver=mysql --mysql-user=root --mysql-password=mysql --test=oltp --mysql-table-engine=${engine} \
--mysql-socket=${sock} \
--num-threads=50 \
--oltp-table-size=$(( 20000000 / ${try} )) --oltp-range-size=0 \
--oltp-point-selects=0 --oltp-simple-ranges=0 --oltp-sum-ranges=0 --oltp-order-ranges=0 \
--oltp-distinct-ranges=0 --oltp-connect-delay=0 \
--oltp-nontrx-mode=update_key"
for a in `seq ${try}`; do
un=oltp$a;
mysql -u root -pmysql -S "${sock}" -e "drop database $un" || true;
mysql -u root -pmysql -S "${sock}" -e "create database $un";
sysbench ${sysbench_options} --mysql-db=$un --oltp-table-name=$un prepare &
done
wait
for a in `seq ${try}`; do
un=oltp$a;
( sysbench ${sysbench_options} --mysql-db=$un --oltp-table-name=$un run 2>&1 | tee /tmp/sysbench-run.${un}.log ) &
done
No errors or segfaults.
Assumed to be fixed in https://github.com/mysql/mysql-server/commit/8870f822d86644fe4578be7025c76478f42100fa
(release 5.7.13)
[5 Nov 2018 13:54]
MySQL Verification Team
Thank you Mr. Black on informing us that a bug is fixed interim 5.7.7 and 5.7.13. Mr. Smith, if you hit another problem on the POWER CPU, please notify us. However, make sure that your OS and CPU are still supported in the MySQL version that you are testing. ARM64 will soon be supported in MySQL 8.0 .......
Description: sysbench read/write workload (any concurrency, any table size) on POWER8 Ubuntu system will hit asserts: #0 0x00003fffb7a2ef94 in __GI_raise (sig=<optimised out>) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x00003fffb7a317a4 in __GI_abort () at abort.c:89 #2 0x0000000010325eec in ut_dbg_assertion_failed ( expr=0x112f0580 "list.count > 0", file=0x112f0590 "/home/stewart/mysql-5.7.7-rc/storage/innobase/include/ut0lst.h", line=279) at /home/stewart/mysql-5.7.7-rc/storage/innobase/ut/ut0dbg.cc:67 #3 0x0000000010d7da8c in ut_list_remove<ut_list_base<trx_t, ut_list_node<trx_t> trx_t::*>, GenericGetNode<trx_t> > (list=..., node=..., get_node=...) at /home/stewart/mysql-5.7.7-rc/storage/innobase/include/ut0lst.h:279 #4 0x0000000010d7b404 in ut_list_remove<ut_list_base<trx_t, ut_list_node<trx_t> trx_t::*> > (elem=0x3fffae8711f8, list=...) at /home/stewart/mysql-5.7.7-rc/storage/innobase/include/ut0lst.h:331 #5 trx_commit_in_memory (serialised=true, mtr=0x3fffac27bf30, trx=0x3fffae8711f8) at /home/stewart/mysql-5.7.7-rc/storage/innobase/trx/trx0trx.cc:1963 #6 trx_commit_low (trx=0x3fffae8711f8, mtr=0x3fffac27bf30) at /home/stewart/mysql-5.7.7-rc/storage/innobase/trx/trx0trx.cc:2177 #7 0x0000000010d7ba28 in trx_commit (trx=0x3fffae8711f8) at /home/stewart/mysql-5.7.7-rc/storage/innobase/trx/trx0trx.cc:2201 #8 0x0000000010d7be24 in trx_commit_for_mysql (trx=0x3fffae8711f8) at /home/stewart/mysql-5.7.7-rc/storage/innobase/trx/trx0trx.cc:2431 #9 0x0000000010bcd194 in innobase_commit_low (trx=<optimised out>) at /home/stewart/mysql-5.7.7-rc/storage/innobase/handler/ha_innodb.cc:3612 #10 0x0000000010bde1c4 in innobase_commit (hton=<optimised out>, thd= 0x3fbc7c000b40, commit_trx=<optimised out>) at /home/stewart/mysql-5.7.7-rc/storage/innobase/handler/ha_innodb.cc:3771 #11 0x00000000103b353c in ha_commit_low (thd=0x3fbc7c000b40, all=<optimised out>, run_after_commit=<optimised out>) at /home/stewart/mysql-5.7.7-rc/sql/handler.cc:1719 #12 0x0000000010828f3c in TC_LOG_DUMMY::commit (this=<optimised out>, thd=<optimised out>, all=<optimised out>) at /home/stewart/mysql-5.7.7-rc/sql/log.h:104 #13 0x00000000103b42bc in ha_commit_trans (thd=0x3fbc7c000b40, all=<optimised out>, ignore_global_read_lock=<optimised out>) at /home/stewart/mysql-5.7.7-rc/sql/handler.cc:1615 (gdb) down #5 trx_commit_in_memory (serialised=true, mtr=0x3fffac27bf30, trx=0x3fffae8711f8) at /home/stewart/mysql-5.7.7-rc/storage/innobase/trx/trx0trx.cc:1963 1963 UT_LIST_REMOVE(trx_sys->rw_trx_list, trx); (gdb) list 1958 1959 trx_sys_mutex_enter(); 1960 1961 check_trx_state(trx); 1962 1963 UT_LIST_REMOVE(trx_sys->rw_trx_list, trx); 1964 1965 ut_d(trx->in_rw_trx_list = false); 1966 1967 MONITOR_INC(MONITOR_TRX_RW_COMMIT); How to repeat: Build MySQL 5.7.7 on Ubuntu on POWER, run sysbench read/write. Suggested fix: I haven't root caused this yet. Read/write workloads seems fine (any concurrency... 2048 threads is fine over hundreds of hardware threads). Read/write though... something goes horribly wrong.