Bug #48130 Expected failures should not count towards max-test-fail
Submitted: 17 Oct 2009 16:16 Modified: 18 Dec 2009 23:41
Reporter: Bjørn Munch Email Updates:
Status: Closed Impact on me:
None 
Category:Tools: MTR / mysql-test-run Severity:S3 (Non-critical)
Version:5.1+ OS:Any
Assigned to: Bjørn Munch CPU Architecture:Any
Tags: mtr

[17 Oct 2009 16:16] Bjørn Munch
Description:
With many tests marked as "expected", test batches tend to get aborted way too early. Only non-expected ("red") failures should count towards the maximum allowed before mtr aborts.

The current behaviour can have the counter-intuitive effect of a bux fix causing *more* tests to fail. This has been seen to happen when some tests marked as ex-fail no longer fail, so that MTR gets ro run more tests (which may fail unexpectedly) before giving up.

How to repeat:
Run a test batch with many tests listed in the experimental list, and with some of those failing.
[17 Oct 2009 16:35] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/87233

2847 Bjorn Munch	2009-10-17
      Bug #48130 Expected failures should not count towards max-test-fail
      Test batches may be terminated too early
      Avoid counting exp-fail tests
[19 Oct 2009 13:49] Bjørn Munch
Pushed to 5.1-mtr, trunk-mtr, next-mr-mtr, 6.0-codebase-mtr
[22 Oct 2009 20:17] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091022201524-0efl2fzebfuuf0vk) (version source revid:bjorn.munch@sun.com-20091019132047-z83ij5ddiu7j34hu) (merge vers: 6.0.14-alpha) (pib:13)
[22 Oct 2009 20:18] Bugs System
Pushed into 5.5.0-beta (revid:alik@sun.com-20091022201318-jfvtrzd6lb07cwp5) (version source revid:bjorn.munch@sun.com-20091019131216-hx3jalmqpyd2j82n) (merge vers: 5.5.0-beta) (pib:13)
[22 Oct 2009 23:22] Paul DuBois
Test suite change. No changelog entry needed.

Setting report to NDI pending push into 5.1.x.
[23 Oct 2009 7:34] Bugs System
Pushed into 5.1.41 (revid:bjorn.munch@sun.com-20091021073307-ummbh6668hvfxqjv) (version source revid:bjorn.munch@sun.com-20091021073307-ummbh6668hvfxqjv) (merge vers: 5.1.41) (pib:13)
[23 Oct 2009 15:18] Paul DuBois
Test suite change. No changelog entry needed.
[18 Dec 2009 10:38] Bugs System
Pushed into 5.1.41-ndb-7.1.0 (revid:jonas@mysql.com-20091218102229-64tk47xonu3dv6r6) (version source revid:jonas@mysql.com-20091218095730-26gwjidfsdw45dto) (merge vers: 5.1.41-ndb-7.1.0) (pib:15)
[18 Dec 2009 10:53] Bugs System
Pushed into 5.1.41-ndb-6.2.19 (revid:jonas@mysql.com-20091218100224-vtzr0fahhsuhjsmt) (version source revid:jonas@mysql.com-20091217101452-qwzyaig50w74xmye) (merge vers: 5.1.41-ndb-6.2.19) (pib:15)
[18 Dec 2009 11:08] Bugs System
Pushed into 5.1.41-ndb-6.3.31 (revid:jonas@mysql.com-20091218100616-75d9tek96o6ob6k0) (version source revid:jonas@mysql.com-20091217154335-290no45qdins5bwo) (merge vers: 5.1.41-ndb-6.3.31) (pib:15)
[18 Dec 2009 11:22] Bugs System
Pushed into 5.1.41-ndb-7.0.11 (revid:jonas@mysql.com-20091218101303-ga32mrnr15jsa606) (version source revid:jonas@mysql.com-20091218064304-ezreonykd9f4kelk) (merge vers: 5.1.41-ndb-7.0.11) (pib:15)