Bug #82615 Need a way to skip flaky tests

• Submitted: 17 Aug 2016 14:32
• Modified: 27 Apr 2017 15:59
• Reporter: Steinar Gunderson
• Status: Closed
• Category: Tools: MTR / mysql-test-run
• Severity: S3 (Non-critical)
• Version: 8.0.1-dmr
• OS: Any
• CPU Architecture: Any

[17 Aug 2016 14:32] Steinar Gunderson

Description:
Hi,

It would be nice with some option in MTR to allow repeats of tests that fail, and then return a success error code if they pass at least once. We have many flaky tests, and when we run almost 5000 of them, one of them is pretty much bound to fail due to chance. This is suboptimal if we want to enforce build/test success before submit, and it teaches people to just ignore the continuous build.

Currently I run with --retry=3 --retry-failure=3 (what is the difference between those two options, anyway? --help is not very useful), and then I get things like:

rpl_gtid.rpl_gtid_temp_table 'mix'       [ skipped ]  Doesn't support --binlog-format='mixed'
[...]
Retrying test rpl_gtid.rpl_gtid_temp_table, attempt(2/3)...
rpl_gtid.rpl_gtid_temp_table 'row'       w7 [ retry-pass ]   3602
Retrying test rpl_gtid.rpl_gtid_temp_table, attempt(3/3)...
rpl_gtid.rpl_gtid_temp_table 'row'       w7 [ retry-pass ]   2907

Failing test(s): rpl_gtid.rpl_gtid_temp_table

which means that the test failed even though it passed in 2/3 cases.

How to repeat:
-

Suggested fix:
Make an option to:

- Stop trying once we get a retry-pass (ie., drop attempt 3/3 above).
- Mark clearly in the end summary that rpl_gtid.rpl_gtid_temp_table isn't failing, it's flaky.
- Return a different status code if no tests were hard-failing (only flaky), so a continuous build could detect that and return success. If that's not possible, simply return success.

[27 Apr 2017 15:59] Paul DuBois

Posted by developer:
 
Noted in 8.0.2 changelog.

mysql-test-run.pl now supports an --report-unstable-tests option that
has these effects:

* Reports any test that has passed using at least one retry attempt in
  a separate category called "Unstable tests" in the summary.

* If all failures are due to unstable tests, mysql-test-run.pl produces
  a warning but exits successfully.

* Adds a new XML tag to report unstable tests, if the --xml-report
  option is also specified.