Tests were run on a 16-core Xeon Sun Fire X4450 system running Solaris Nevada build 106 and MySQL 5.40-Beta. Tests were run under the following conditions: Baseline: Default behavior, TS scheduling class, MySQL thread scheduling Test 1: TS scheduling class, --skip thread-priority Test 2; mysqld started with in FX scheduling class w/ priority 59 and --skip-thread-priority Test 3 results were within 4% of the baseline on Sysbench Read-Only and 10% on Read-Write. In both cases Test 3 gave the best results. Test 2 results were more complicated. On Sysbench Read-Only, Test 2 had better results up 16 threads. At 32 threads it had only 81% the performance of the baseline. At 128 threads Test 2 performance was only 60% of the baseline. On Sysbench Read-Write, Test2 was about the same as the baseline from 1-16 threads. At 32 threads it had 82% the performance of the baseline and at 128 threads only 34% the performance of the baseline. Sysbench Comparisons Rd-Only TS-STP FXpri59 Thrd /TS /TS 1 1.03 1.02 2 1.05 1.05 4 1.06 1.04 8 1.04 1.05 12 1.02 1.04 16 1.14 1.05 32 0.81 1.05 48 0.77 1.05 64 0.77 1.04 80 0.70 1.04 96 0.69 1.03 112 0.69 1.03 128 0.60 1.04 Rd-Wrt TS_STP FXpri59 Thrd /TS /TS 1 1.04 1.05 2 1.04 1.05 4 1.04 1.04 8 0.99 1.04 12 1.01 1.05 16 1.00 1.04 32 0.82 1.07 48 0.69 1.09 64 0.57 1.08 80 0.49 1.09 96 0.44 1.09 112 0.39 1.10 128 0.34 1.10 I then executed 32-thread Sysbench Rd-only tests and gathered prstat and plockstat data. What it shows is that per thread time waiting on User Lock (LCK) increases from .3% with default to 15% with TS --skip-thread-priority. plockstat shows a corresponding increase in time spent waiting on LOCK_open in open_table and close_open_tables: MySQL 5.4.0-Beta TS PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 11101 mysql 35 8.5 0.0 0.0 0.0 0.3 46 9.4 .7M 655 .76 0 mysqld/39 11101 mysql 35 8.5 0.0 0.0 0.0 0.3 46 9.5 .7M 672 .76 0 mysqld/50 11101 mysql 35 8.4 0.0 0.0 0.0 0.3 47 8.9 .7M 718 .75 0 mysqld/45 11101 mysql 35 8.4 0.0 0.0 0.0 0.3 47 9.3 .7M 556 .75 0 mysqld/36 11101 mysql 35 8.4 0.0 0.0 0.0 0.3 47 9.5 .7M 647 .75 0 mysqld/34 11101 mysql 35 8.4 0.0 0.0 0.0 0.3 47 9.2 .7M 640 .75 0 mysqld/44 11101 mysql 35 8.3 0.0 0.0 0.0 0.3 47 10 .7M 621 .74 0 mysqld/56 11101 mysql 35 8.3 0.0 0.0 0.0 0.3 48 9.0 .7M 706 .74 0 mysqld/47 11101 mysql 34 8.4 0.0 0.0 0.0 0.3 48 9.0 .7M 1K .74 0 mysqld/38 11101 mysql 34 8.4 0.0 0.0 0.0 0.3 47 9.9 .7M 599 .74 0 mysqld/37 11101 mysql 34 8.3 0.0 0.0 0.0 0.3 47 9.7 .7M 570 .74 0 mysqld/49 11101 mysql 34 8.3 0.0 0.0 0.0 0.3 48 8.9 .7M 627 .74 0 mysqld/46 11101 mysql 34 8.3 0.0 0.0 0.0 0.3 48 9.2 .7M 658 .74 0 mysqld/51 11101 mysql 34 8.3 0.0 0.0 0.0 0.3 48 9.1 .7M 600 .74 0 mysqld/55 11101 mysql 34 8.3 0.0 0.0 0.0 0.3 48 8.8 .7M 621 .74 0 mysqld/33 Total: 1 processes, 56 lwps, load averages: 42.79, 26.39, 11.66 0 Mutex block Count nsec Lock Caller ------------------------------------------------------------------------------- 90349 112707 mysqld`LOCK_open mysqld`__1cRclose_open_tables6FpnDTHD__v_+0x1e 64600 127979 mysqld`LOCK_open mysqld`__1cKopen_table6FpnDTHD_pnKTABLE_LIST_pnLst_mem_root_pbI_pnIst_table__+0x49d 4357 194186 mysqld`.XAB_pdeMOT2JECD.LOCK_alarm mysqld`thr_end_alarm+0x17 4129 180325 mysqld`.XAB_pdeMOT2JECD.LOCK_alarm mysqld`thr_alarm+0x37 3250 202073 mysqld`LOCK_thread_count mysqld`__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_+0x7b 3007 184848 mysqld`LOCK_thread_count mysqld`__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_+0xfb1 MySQL 5.4.0-Beta TS --skip-thread-priority Total: 1 processes, 56 lwps, load averages: 4.84, 17.29, 10.38 PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID 11251 mysql 31 8.0 1.1 0.0 0.0 15 3.7 41 .7M .6M .37 0 mysqld/39 11251 mysql 31 8.1 1.1 0.0 0.0 15 3.8 41 .7M .6M .37 0 mysqld/57 11251 mysql 31 7.9 1.1 0.0 0.0 15 4.0 41 .7M .6M .37 0 mysqld/52 11251 mysql 31 8.0 1.0 0.0 0.0 15 3.9 41 .7M .5M .37 0 mysqld/55 11251 mysql 31 8.0 1.1 0.0 0.0 15 3.6 41 .7M .6M .37 0 mysqld/28 11251 mysql 31 8.0 1.1 0.0 0.0 15 3.6 41 .7M .5M .37 0 mysqld/34 11251 mysql 31 8.0 1.0 0.0 0.0 15 3.8 41 .7M .6M .37 0 mysqld/26 11251 mysql 31 8.0 1.1 0.0 0.0 15 3.7 41 .7M .6M .37 0 mysqld/33 11251 mysql 31 7.8 1.1 0.0 0.0 15 3.6 41 .7M .6M .37 0 mysqld/32 11251 mysql 31 8.0 1.1 0.0 0.0 15 3.6 41 .7M .6M .37 0 mysqld/45 11251 mysql 31 7.9 1.1 0.0 0.0 15 3.6 41 .7M .6M .37 0 mysqld/53 11251 mysql 31 7.9 1.1 0.0 0.0 15 3.5 41 .7M .6M .37 0 mysqld/37 11251 mysql 30 7.9 1.1 0.0 0.0 15 3.8 41 .7M .6M .37 0 mysqld/42 11251 mysql 30 8.1 1.1 0.0 0.0 15 3.6 41 .7M .6M .36 0 mysqld/40 11251 mysql 31 7.8 1.1 0.0 0.0 15 3.7 41 .7M .6M .37 0 mysqld/51 Total: 1 processes, 56 lwps, load averages: 29.68, 24.80, 15.56 0 Mutex block Count nsec Lock Caller ------------------------------------------------------------------------------- 1706912 397251 mysqld`LOCK_open mysqld`__1cKopen_table6FpnDTHD_pnKTABLE_LIST_pnLst_mem_root_pbI_pnIst_table__+0x49d 1385503 406732 mysqld`LOCK_open mysqld`__1cRclose_open_tables6FpnDTHD__v_+0x1e 600011 295862 mysqld`LOCK_thread_count mysqld`__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_+0xfb1 496585 297221 mysqld`LOCK_thread_count mysqld`__1cQdispatch_command6FnTenum_server_command_pnDTHD_pcI_b_+0x7b 461723 273930 0x13ca2b0a0 mysqld`thr_lock+0x4a 451048 274544 mysqld`.XAB_pdeMOT2JECD.LOCK_alarm mysqld`thr_alarm+0x37 339003 276760 mysqld`THR_LOCK_time mysqld`my_micro_time_and_time+0x17 My theory is that threads are being preempted while holding LOCK_open in open_table and close_open_tables. I reran 32 thread sysbench read-only and ran a D script which aggregates the number of times the functions are executed and the number of times mysqld is in those functions when a thread is preempted. What it shows is running with --skip-priority resulted in a 458x increase in number of times mysqld was in open_table when preempted and a 478x increase in number of times mysqld was in close_open_tables when preempted. MySQL 5.4.0-Beta TS Number times function executed __1cKopen_table6FpnDTHD_pnKTABLE_LIST_pnLst_mem_root_pbI_pnIst_table__ 22410523 __1cRclose_open_tables6FpnDTHD__v_ 22410523 Number times preempted when in function __1cRclose_open_tables6FpnDTHD__v_ 1461 __1cKopen_table6FpnDTHD_pnKTABLE_LIST_pnLst_mem_root_pbI_pnIst_table__ 2299 MySQL 5.4.0-Beta TS --skip-thread-priority Number times function executed __1cKopen_table6FpnDTHD_pnKTABLE_LIST_pnLst_mem_root_pbI_pnIst_table__ 17641843 __1cRclose_open_tables6FpnDTHD__v_ 17641843 Number times preempted when in function __1cRclose_open_tables6FpnDTHD__v_ 697015 __1cKopen_table6FpnDTHD_pnKTABLE_LIST_pnLst_mem_root_pbI_pnIst_table__ 1052638 I then did an experiment where I bracketed the calls to pthread_mutex_lock(&LOCK_open)/pthread_mutex_unlock(&LOCK_open) with schedctl_start/schedctl_stop. schedctl is used to hint to Solaris that preemptions of that thread should be avoided. In a 32-thread sysbench read-only test the number of preemptions was reduced and resulted in a 1.28x increase in TPS. MySQL 5.4.0-Beta --skip-thread-priority with schedctl Number times function executed __1cKopen_table6FpnDTHD_pnKTABLE_LIST_pnLst_mem_root_pbI_pnIst_table__ 24082669 __1cRclose_open_tables6FpnDTHD__v_ 24082669 Number times preempted when in function __1cRclose_open_tables6FpnDTHD__v_ 2233 __1cKopen_table6FpnDTHD_pnKTABLE_LIST_pnLst_mem_root_pbI_pnIst_table__ 367155 Complete sysbench tests were run with schedctl. Although results were consistently better than --skip-thread-priority, they were still 10-20% worse than running with thread scheduling. The best case for --skip-thread-priority is running with mysqld in FX scheduling class and priority 59.