MySQL Bugs: #43917: MTR2 does not report accurate test statistics when using the 'repeat=n' option

Bug #43917	MTR2 does not report accurate test statistics when using the 'repeat=n' option
Submitted:	27 Mar 2009 17:36	Modified:	29 May 2009 1:11
Reporter:	Patrick Crews	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tools: MTR / mysql-test-run	Severity:	S3 (Non-critical)
Version:	5.1+	OS:	Any
Assigned to:	Bjørn Munch	CPU Architecture:	Any
Tags:	MTR2, mysql-test-run, repeat, test statistics

Description:
When using MTR2's --repeat=n option, the program does not report accurate test-run statistics.

As an example, I was testing a patch to innodb.test (I have attached the .test and .result files for testing / verification purposes - you will need to check disabled.def to ensure the test isn't disabled).

When I ran
./mtr --force --repeat=10 innodb 

I received the following output:
Failed 1/1 tests, 0.00% were successful.

Failing test(s): main.innodb

This was despite the fact that 5/10 iterations of the test passed.

How to repeat:
Using the innodb.test and innodb.result files I have attached as a tarball, run the following:

./mtr --force --repeat=10 innodb

Observe the output.  You might need to re-enable the test in t/disabled.def to get it to run as it is currently disabled for a bug.

With the attached files, the test *should* pass every other time, but you will receive end-test output as I have noted in the Description section.

Suggested fix:
This could be tricky, but I think the current reporting is inaccurate.

If I run a test 10 times and half of the runs fail, counting the run as consisting of a single test is not accurate.  This is especially problematic when trying to recreate some rare test bugs (say running a test 10,000 times).

Personally, I think each run of a test (not counting fail-retries) should count as a 'test' and the end report of my example run should be Failed 5/10 test, 50.00% were successful.

We might also expand reporting to explicitly state that a test was a repeat, but that could become problematic if --repeat'ing more than one test (reporting could become bloated / less efficient).

Example .test and .result file to exhibit the bug behavior

Attachment: mtr_example.tar (application/x-tar, text), 180.00 KiB.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/70887

2776 Bjorn Munch	2009-03-31
      Bug #43917 MTR2 does not report accurate test statistics when using the 'repeat=n' option
      In practice, only the last run of the test was counted
      Add a separate counter rep_failures for failures before last run
      modified:
        mysql-test/lib/mtr_report.pm
        mysql-test/mysql-test-run.pl

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/71069

2779 Bjorn Munch	2009-04-01
      Bug #43917 MTR2 does not report accurate test statistics when using the 'repeat=n'
      option
      In practice, only the last run of the test was counted
      Add a separate counter rep_failures for failures before last run
      modified:
        mysql-test/lib/mtr_report.pm
        mysql-test/mysql-test-run.pl

Pushed to 6.0-mtr and 5.1-mtr

Pushed into 6.0.12-alpha (revid:alik@sun.com-20090514161729-xiup7q8jlu1zfy7s) (version source revid:bjorn.munch@sun.com-20090401120042-87osxxgrqyypyfnz) (merge vers: 6.0.11-alpha) (pib:6)

Test suite changes. No changelog entry needed.

Setting report to NDI pending push into 5.1.x.

Pushed into 5.1.36 (revid:joro@sun.com-20090528073639-yohsb4q1jzg7ycws) (version source revid:bjorn.munch@sun.com-20090525205831-zitq9aysxy5zgf5r) (merge vers: 5.1.36) (pib:6)

Test suite change. No changelog entry needed.

Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)

Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)

Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)