Bug #29921 | MYSQLparse uses 3X more CPU in MySQL 5.0 than yyparse in MySQL 4.0 | ||
---|---|---|---|
Submitted: | 20 Jul 2007 3:39 | Modified: | 4 Mar 2008 22:29 |
Reporter: | Mark Callaghan | Email Updates: | |
Status: | Closed | Impact on me: | |
Category: | MySQL Server: Parser | Severity: | S5 (Performance) |
Version: | 5.0.37 | OS: | Any |
Assigned to: | Marc ALFF | CPU Architecture: | Any |
Tags: | bfsm_2007_08_02, bfsm_2007_10_18, Contribution, parser, performance, regression |
[20 Jul 2007 3:39]
Mark Callaghan
[20 Jul 2007 3:41]
Mark Callaghan
This is oprofile output for the benchmark using InnoDB tables and MySQL 4.0.26. I have listed the top 30 functions: samples % app name symbol name 20790098 9.6387 mysqld yyparse() 13978509 6.4807 mysqld yylex(void*) 12371259 5.7355 libc-2.3.5.so memcpy 9134302 4.2348 mysqld cmp_dtuple_rec_with_match 6339029 2.9389 mysqld buf_calc_page_new_checksum 5467357 2.5348 mysqld log_group_write_buf 5280038 2.4479 mysqld btr_search_guess_on_hash 5228149 2.4239 mysqld mysql_parse(THD*, char*, unsigned int) 4697479 2.1778 mysqld page_cur_insert_rec_write_log 4529755 2.1001 mysqld Item_int::save_in_field(Field*, bool) 3836652 1.7787 libpthread-2.3.5.so __pthread_mutex_unlock_usercnt 3082379 1.4290 mysqld setup_fields(THD*, st_table_list*, List<Item>&, bool, List<Item>*, bool) 2903332 1.3460 mysqld btr_cur_optimistic_insert 2814160 1.3047 libpthread-2.3.5.so pthread_getspecific 2799271 1.2978 mysqld page_cur_search_with_match 2710013 1.2564 mysqld Item_string::save_in_field(Field*, bool) 2593889 1.2026 mysqld row_insert_for_mysql 2593100 1.2022 mysqld Item_string::~Item_string() 2545890 1.1803 mysqld page_cur_insert_rec_low 2434105 1.1285 mysqld page_rec_get_next 2414043 1.1192 mysqld Item_int::~Item_int() 2340320 1.0850 libc-2.3.5.so __GI_____strtoll_l_internal 2238759 1.0379 libpthread-2.3.5.so pthread_mutex_trylock 2137703 0.9911 mysqld btr_cur_search_to_nth_level 2092474 0.9701 mysqld rec_get_nth_field 2048178 0.9496 mysqld mtr_commit 2002929 0.9286 mysqld alloc_root
[20 Jul 2007 3:43]
Mark Callaghan
These are the top 30 from MySQL 5.0.37 using InnoDB tables: Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000 samples % app name symbol name 67093324 19.4305 mysqld MYSQLparse(void*) 14889263 4.3120 mysqld rec_get_offsets_func 12838530 3.7181 libc-2.3.5.so memcpy 12018824 3.4807 mysqld log_group_write_buf 9654171 2.7959 mysqld MYSQLlex(void*, void*) 7601190 2.2013 mysqld cmp_dtuple_rec_with_match 7159070 2.0733 libc-2.3.5.so strstr 6870511 1.9897 libpthread-2.3.5.so __pthread_mutex_unlock_usercnt 5872749 1.7008 mysqld buf_calc_page_new_checksum 5670346 1.6422 mysqld get_text(st_lex*) 5587272 1.6181 mysqld btr_search_guess_on_hash 5515860 1.5974 mysqld alloc_root 4594264 1.3305 mysqld Query_arena::free_items() 4551895 1.3182 mysqld Item_int::save_in_field(Field*, bool) 4479174 1.2972 mysqld copy_and_convert(char*, unsigned int, charset_info_st*, char const*, unsigned int, charset_info_st*, unsigned int*) 4186962 1.2126 mysqld page_cur_insert_rec_low 4151865 1.2024 mysqld Item_string::save_in_field(Field*, bool) 3901579 1.1299 libpthread-2.3.5.so pthread_mutex_trylock 3822935 1.1071 mysqld Item::used_tables() const 3753735 1.0871 mysqld setup_fields(THD*, Item**, List<Item>&, bool, List<Item>*, bool) 3544822 1.0266 mysqld page_cur_search_with_match 3405702 0.9863 mysqld mem_area_free 3388987 0.9815 mysqld Item::check_cols(unsigned int) 3334453 0.9657 libpthread-2.3.5.so pthread_mutex_unlock 3290290 0.9529 mysqld my_uni_utf8 3199040 0.9265 mysqld fill_record(THD*, Field**, List<Item>&, bool) 3169733 0.9180 mysqld buf_page_get_gen
[20 Jul 2007 3:44]
Mark Callaghan
These are the top 30 from MySQL 5.0.37 with system_charset_info changed to use latin1 rather than utf8: Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000 samples % app name symbol name 40761850 19.4885 mysqld MYSQLparse(void*) 9352251 4.4714 mysqld rec_get_offsets_func 8789276 4.2022 libc-2.3.5.so memcpy 8090572 3.8682 mysqld log_group_write_buf 5784592 2.7656 mysqld MYSQLlex(void*, void*) 4662155 2.2290 mysqld cmp_dtuple_rec_with_match 4392068 2.0999 libc-2.3.5.so strstr 4300587 2.0561 mysqld get_text(st_lex*) 4191729 2.0041 libpthread-2.3.5.so __pthread_mutex_unlock_usercnt 3746288 1.7911 mysqld buf_calc_page_new_checksum 3337162 1.5955 mysqld alloc_root 3009014 1.4386 mysqld btr_search_guess_on_hash 2867279 1.3709 mysqld Query_arena::free_items() 2716966 1.2990 libpthread-2.3.5.so pthread_mutex_trylock 2614492 1.2500 mysqld page_cur_search_with_match 2557889 1.2229 mysqld Item_string::save_in_field(Field*, bool) 2511082 1.2006 mysqld page_cur_insert_rec_low 2503175 1.1968 mysqld Item_int::save_in_field(Field*, bool) 2385286 1.1404 mysqld Item::used_tables() const 2231738 1.0670 mysqld setup_fields(THD*, Item**, List<Item>&, bool, List<Item>*, bool) 2155671 1.0306 mysqld mem_area_free 2104924 1.0064 mysqld buf_page_get_gen 1927640 0.9216 mysqld Item::check_cols(unsigned int) 1890692 0.9040 mysqld fill_record(THD*, Field**, List<Item>&, bool) 1725761 0.8251 libpthread-2.3.5.so pthread_getspecific 1715146 0.8200 mysqld page_cur_insert_rec_write_log 1709014 0.8171 mysqld btr_cur_search_to_nth_level
[20 Jul 2007 14:09]
Mark Callaghan
The test was 4% faster after I changed system_charset_info to use latin1 (the default database charset) rather than utf8. That is only a small part of the problem.
[20 Jul 2007 14:50]
MySQL Verification Team
See bug: http://bugs.mysql.com/bug.php?id=11605
[20 Jul 2007 23:23]
Mark Callaghan
I reran the database reload tests using MyISAM tables. The slowdown is bad. The table below lists the time to finish loading data that uses ~100GB for InnoDB tables. Tests were run for 1, 2, 4, 8 and 16 concurrent sessions inserting into ~400 tables. Sessions did not insert into the same tables and all table types were MyISAM. The server had 4 CPU cores and 12 disks with SW RAID 0. Times are in seconds. Clients and server are on the same host. number of concurrent insert sessions version 1 2 4 8 16 4.0.26 8831 4631 3007 3082 3174 5.0.37 14008 7373 4957 5582 5408 This regression is worse than 50%.
[21 Jul 2007 4:36]
Mark Callaghan
This is a simple test case. MySQL 5.0.37 does this in 48 seconds. MySQL 4.0.26 does this in 31 seconds. This is on a 2.4GHz Opteron. drop table if exists t1; create table t1 (i int) type = myisam; insert into t1 values (1); update t1 set i = 2; <repeat previous line 1M times> ------------ Another simple test takes 30 seconds for MySQL 4 and 44 seconds for MySQL 5 drop table if exists i; create table i (i int) type = myisam; insert into i values (1); <repeat 1M times>
[21 Jul 2007 18:15]
Mark Callaghan
I tried removing many of the new features from the parser for 5.0. This includes: triggers, stored procedures and views. The parser was not any faster with support for those features removed. I then used rdtsc to determine what part of MYSQLparse/yyparse was slower in 5.0. The biggest slowdown is parsing the VALUES list for the insert statment. For each entry in the VALUES list in the 'insert into Foo values (....)' statement, the expr token uses ~5000 cycles in MySQL 4 and ~7500 cycles in MySQL 5. This is the primary factor in the 50% slowdown in MySQL 5 The expr nonterminal in MySQL 4 is simple: expr: expr_expr {$$ = $1; } | simple_expr {$$ = $1; }; The expr nonterminal in MySQL 5 is complex. Notice all of the memory alloc/dealloc operations and other work that includes in the common case: new List, push_front call, pop call, delete list. expr: bool_term { Select->expr_list.push_front(new List<Item>); } bool_or_expr { List<Item> *list= Select->expr_list.pop(); if (list->elements) { list->push_front($1); $$= new Item_cond_or(*list); /* optimize construction of logical OR to reduce amount of objects for complex expressions */ } else $$= $1; delete list; } ;
[21 Jul 2007 19:52]
Mark Callaghan
I changed the values terminal in sql_yacc.yy to use simple_expr and avoid the overhead of expr mentioned previously. This was done to determine whether the overhead of expr is what makes MySQL 5 slower. The new definition of values is: values: values ',' simple_expr { if (Lex->insert_list->push_back($3)) YYABORT; } | simple_expr { if (Lex->insert_list->push_back($1)) YYABORT; } ; I then ran this SQL to compare MySQL 4 with MySQL 5 modifed as described above: drop table if exists i; create table i (i int) type = myisam; insert into i values (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (1),(2),(3),(4),(5),(6),(7),(8),(9),(10), (1),(2),(3),(4),(5),(6),(7),(8),(9),(10); <repeat insert statement 1M times> The time to run this was: * MySQL 4 -> 32 seconds * MySQL 5 original -> 39 seconds * MySQL 5 modified -> 32 seconds So expr in sql_yacc.yy for MySQL 5 is a big part of the regression.
[21 Jul 2007 20:01]
Mark Callaghan
The bool_term nonterminal has the same problem as the expr terminal. Both allocate a List, push it onto another data structure, pop it, and then deallocate the list. Unfortunately, all of this must be done to parse a literal. This is very expensive.
[21 Jul 2007 20:32]
MySQL Verification Team
Thank you for the bug report.
[21 Jul 2007 23:59]
Mark Callaghan
In MySQL 5.0.37, I changed the definition of the values nonterminal in sql_yacc.yy to use simple_expr rather than expr_or_default. This is incorrect, but allows me to avoid the overhead in expr and bool_term. I reran the database reload tests. The change makes performance for MySQL 5.0 much better for the tests with low concurrency (1 or 2 sessions). Performance improvement for 4 or more concurrent sessions does not improve as much. I think this is because I have saturated the CPU on the server -- it has 4 CPU cores. '5.0.37 hack' has the change to sql_yacc.yy. This lists the number of seconds to reload about 100GB of data (when measured in InnoDB datafiles) into MyISAM tables. number of concurrent insert sessions version 1 2 4 8 16 4.0.26 8831 4631 3007 3082 3174 5.0.37 orig 14008 7373 4957 5582 5408 5.0.37 hack 10760 5917 4699 5513 5234
[22 Jul 2007 8:24]
Konstantin Osipov
Mark, this is an excellent analysis. We've had observed the regression internally but never got a change to analyze the cause.
[22 Jul 2007 16:57]
Mark Callaghan
Thanks. My hope for an easy fix might not come true. I changed the grammar to not optimize for expressions with many AND and OR terms and to use the 2 arg constructor for Item_cond_or and Item_cond_and. This change removed the extra new/delete calls. But the performance after this change was not that much better. The grammar for expressions is very different in MySQL 5 than in MySQL 4. There is much more nesting to reach simple_expr. Perhaps changing the grammar to be more similar to MySQL 4 will get back the performance.
[3 Aug 2007 23:29]
Marc ALFF
Hi Mark Many thanks for your report and analysis. I suspect that many different factors have an effect on the performance degradation, so to address this in general, I would like to propose to: - keep bug#29921 open, and use it to collect observations / comments - fix each issue found independently, with a dedicated bug report. This would make the fix/review/merge process much easier Below are some notes of various aspects identified so far, that I plan to work on. Regards, -- Marc ANALYSIS / things to investigate Grammar complexity The number of rules / terminals / non terminals, etc has increased between 4.1 and 5.0. See related comments in bug#11605. A direct effect of this is that more reduces are needed to parse the same construct, causing bison to execute more loops internally. Impact: to assess, I assume it's very low. Possible mitigation: Flatten the rules, to avoid unnecessary R1->R2->R3 reduces, where R1->R3 can be done instead. Risk: Make sure operators priority are not affected when flattening the rules. Actions complexity The issue with the lists used in the AND and OR boolean expressions falls into this category. This specific problem will be fixed with Bug#30237, but other similar issues could also exist. Action: code review of actions in general Impact: localized changes expected, low Memory allocation There is a difference between : - new Item() - new (THD->mem_root) Item() that could have an impact here, since the later does not need to get to thread local storage again and again. Action: code cleanup, mostly. Performance impact to assess Impact of new features In general, constructs like VIEW, TRIGGER, PROCEDURE, FUNCTION etc do affect a lot the grammar with new statements, but should not affect existing statements that do not use the feature. Stored procedures have an effect of changing THD::mem_root during parsing, but since this code is not involved when parsing regular statements, there should be no impact at all. To investigate: the only known side effect of having new statements supported but unused in the parser is the total size of the parser state automaton. There is a concern that at some point increasing this size could cause caching faults, but it's not verified: it's only an hypothesis. Impact of grammar ambiguities Same as above, focused at the following rules: - keyword - keyword_sp Early experiment showed that the parser size is greatly reduced (divided by a factor 3) by removing these rules. The idea is that not having ambiguities with keywords like this should help the parser to find statements faster. Action: assess the real effect on execution time, instead of code size. Parser stack Some rules do use a stack maintained by the mysql code, independently of the stack maintained by bison. See for example the rules: - udf_expr_list - expr_list - ident_list which uses Select->expr_list. Action: investigate, assess performance impact compared to using $$ Expressions Because expressions are used in many places, optimizing parsing of expressions (compared to say DDL statements like CREATE PROCEDURE) should have the most positive impact on overall performances. This area will be investigated first. MYSQLlex Investigate if the lexer itself is involved in the regression. Action: implement the suggestion about comparing charsets by number, not by name
[13 Aug 2007 12:37]
Marko Mäkelä
See also Bug #11604 and Bug #11605. (Sorry for the previous entry, which contained wrong bug numbers.)
[13 Aug 2007 19:04]
Mark Callaghan
I will attach changes to sql_yacc.yy that removed about half of the performance regression from MySQL 4.0. The numbers below are from a benchmark that reports the number of seconds to reload all tables of a large (100GB as InnoDB) database. Values for the 'binary' column: 4_64 - 64-bit build of MySQL 4.0.26 5_orig - 64-bit build of MySQL 5.0.37 with parser performance problem 5_fix - 64-bit build of MySQL 5.0.37 with faster grammar DOP=x lists the number of concurrent sessions. The regression is worst for DOP=1. First, numbers for MyISAM tables. For DOP=1, MySQL5 is 58% slower without the fix and 29% slower with it. binary DOP=1 DOP=2 DOP=4 DOP=8 DOP=16 4_64 10061 5729 3436 3543 3557 5_fix 12995 6995 5369 6378 6493 5_orig 15808 8232 5622 6608 6849 Numbers for InnoDB tables. For DOP=1, MySQL5 is 52% slower without the fix and 30% slower with it. binary DOP=1 DOP=2 DOP=4 DOP=8 DOP=16 4_64 12741 8489 7160 7825 8018 5_fix 16532 10515 8508 9726 9282 5_orig 19315 11744 9368 9859 9401
[13 Aug 2007 19:05]
Mark Callaghan
Patch for changes to sql_yacc.yy in MySQL 5.0.37
Attachment: yacc.c (text/x-csrc), 16.88 KiB.
[17 Aug 2007 18:03]
Marc ALFF
Analysis, grammar complexity Release 4.1: Query | States | Reduces | Select 1 | 43 | 33 | Select 1,2 | 55 | 43 | Select 1,2,3 | 67 | 53 | Increment | +12 | +10 | 'State' counts the number of state transitions executed while parsing the query. 'Reduces' counts the number of rule reductions executed. 'Increment' is the additional cost of having 1 expression Reduces for an expression: NUM_literal: NUM literal: NUM_literal simple_expr: literal expr: simple_expr --> 4 reduces Release 5.0 (5.0-runtime) Query | States | Reduces | Select 1 | 60 | 50 | Select 1,2 | 87 | 75 | Select 1,2,3 | 114 | 100 | Increment | +27 | +25 | Reduces for an expression: NUM_literal: NUM literal: NUM_literal simple_expr: literal factor: simple_expr term: factor value_expr: term bit_factor: value_expr bit_term: bit_factor bit_expr: bit_term predicate: bit_expr bool_pri: predicate bool_test: bool_pri bool_factor: bool_test (reduce action @71): { Select->expr_list.push_front(new List<Item>); } bool_and_expr: /* empty */ bool_term: bool_factor bool_and_expr (reduce action @70): { Select->expr_list.push_front(new List<Item>); } bool_or_expr: /* empty */ expr: bool_term bool_or_expr --> 19 reduces Release 5.0 + bug#30237 (5.0-perf) Query | States | Reduces | Select 1 | 57 | 47 | Select 1,2 | 81 | 69 | Select 1,2,3 | 105 | 91 | Increment | +24 | +22 | Reduces for an expression: NUM_literal: NUM literal: NUM_literal simple_expr: literal factor: simple_expr term: factor value_expr: term bit_factor: value_expr bit_term: bit_factor bit_expr: bit_term predicate: bit_expr bool_pri: predicate bool_test: bool_pri bool_factor: bool_test bool_term: bool_factor bool_xterm: bool_term expr: bool_xterm --> 16 reduces Release 5.1 (5.1-runtime) Query | States | Reduces | Select 1 | 60 | 50 | Select 1,2 | 87 | 75 | Select 1,2,3 | 114 | 100 | Increment | +27 | +25 | Reduces for an expression: Same as 5.0 ---------- The regression in general is caused in part by the number of reduces needed to parse an expression, which increased from 5 to 19 between 4.1 and 5.0. This cause the bison generated code to loop 19 times instead of 4 in this case, and cause the differences in MYSQLparse() seen with oprofile/gprof
[24 Aug 2007 15:33]
Marc ALFF
See related bug#30625
[25 Aug 2007 1:13]
Marc ALFF
Performance update Test case: 1,000,000 inserts of 100 values into a blackhole table. This test forces the evaluation of 100 Milions expressions in the parser, and is designed to expose the parser cost, by removing the cost of the runtime execution of the query (blackhole). All the tests below have been executed on the same hardware, a single CPU amd64 box running Linux. MySQL 4.1.24, gprof Total time: 6m6.915s (367s) time in yyparse(): 48.48s MySQL 5.0.48, gprof Total time: 10m6.592s (607s) time in MYSQLparse(): 125.13s Regression in total time: 240s Regression in MYSQLparse(): 76.65s MySQL 5.0.48 + Bug#30237, gprof Total time: 8m12.277s (492s) time in MYSQLparse(): 97.10s Regression in total time: 125s Regression in MYSQLparse(): 48.62s Based on the total time, the fix for Bug#30237 reclaimed 115 seconds from 240, or around 50 percent. Please keep in mind that: a) while this number is encouraging, it needs to be validated against the original use case, as a verification. b) there is other work in progress (bug#30625) that also improve performances, so there is more to gain. As a preview, prototyping of bug#30625 gives the following results: MySQL 5.0.48 + Bug#30237 + Bug#30625, gprof Total time: 7m31.371 (451s) time in MYSQLparse(): 64.15s This represent a regression of 84 seconds instead of 240, or a reclaim of 65 percents (to confirm). The patch for Bug#30625 is not available yet, it's to be posted soon. The patch for Bug#30237 is already available. Regards, -- Marc
[28 Aug 2007 16:24]
Marc ALFF
See related bug#30625, for which a performance improvement patch is available.
[11 Sep 2007 0:07]
Marc ALFF
GPROF profiling for 4.1
Attachment: perf-4.1.txt.gz (application/x-gzip, text), 127.24 KiB.
[11 Sep 2007 0:08]
Marc ALFF
GPROF profiling for 5.0.45
Attachment: perf-5.0.45.txt.gz (application/x-gzip, text), 144.20 KiB.
[11 Sep 2007 0:08]
Marc ALFF
GPROF profiling for 5.0.50
Attachment: perf-5.0.txt.gz (application/x-gzip, text), 144.70 KiB.
[11 Sep 2007 0:10]
Marc ALFF
time with gprof, 4.1
Attachment: time-4.1-gprof.log (text/x-log), 1.23 KiB.
[11 Sep 2007 0:11]
Marc ALFF
time with gprof, 5.0.45
Attachment: time-5.0.45-gprof.log (text/x-log), 1.38 KiB.
[11 Sep 2007 0:12]
Marc ALFF
time with gprof, 5.0.50
Attachment: time-5.0-gprof.log (text/x-log), 1.35 KiB.
[11 Sep 2007 0:14]
Marc ALFF
time results, 4.1
Attachment: time-4.1.log (text/x-log), 1.28 KiB.
[11 Sep 2007 0:14]
Marc ALFF
time results, 5.0.45
Attachment: time-5.0.45.log (text/x-log), 1.43 KiB.
[11 Sep 2007 0:15]
Marc ALFF
time results, 5.0.50
Attachment: time-5.0.log (text/x-log), 1.43 KiB.
[11 Sep 2007 0:26]
Marc ALFF
test 1,000,000 inserts of 100 values in a blackhole table, all performed on the same hardware (dual core pentium64) Timing under gprof 4.1.24: 6m46 (406 s) 5.0.45: 10m23 (623 s) 5.0.50: 8m50 (530 s) Time spent in MYSQLparse() / yyparse() 4.1.24: 25.21 s 5.0.45: 68.82 s 5.0.50: 45.01 s Timing under a regular build (no profiling) 4.1.24: 5m58 (358 s) 5.0.45: 7m25 (445 s) 5.0.50: 5m48 (348 s)
[11 Sep 2007 0:47]
Marc ALFF
Hi Mark Please see the new performance numbers (previous comments in the bug report), obtained in a test environment (focused on stressing the parser). Could you try MySQL 5.0.50 (from the BK repository) or later, with the load data benchmark that was used when reporting the bug, and inform us of the changes in performances ? Thanks a lot for your feedback. Regards, -- Marc
[17 Sep 2007 19:05]
Mark Callaghan
I will do this soon. I was distracted by http://bugs.mysql.com/bug.php?id=30738
[27 Sep 2007 17:35]
Mark Callaghan
Can you provide a patch file with only your changes? I need to push this into 5.0.37 for testing and the diff of sql_yacc.yy between 5.0.37 and 5.0.50 (from bk) has too many changes.
[5 Oct 2007 22:04]
Marc ALFF
See related bug#17229, which is another symptom of the same issue.
[4 Mar 2008 21:52]
Marc ALFF
See related blog: http://mysqlha.blogspot.com/2008/02/is-mysql-getting-faster.html
[4 Mar 2008 22:29]
Marc ALFF
Several performance related issues have been fixed in the parser: - Bug#30237 (Performance regression in boolean expressions) - Bug#30333 (Performance, expressions lists in the parser) - Bug#30625 (Performance, reduce depth for expressions) The performance degradation reported has been addressed, so this bug is now closed. Feel free to re-open this bug report, with new performance measures, if further improvement is needed.