MySQL Bugs: #42888: Support for flexible test suites

Bug #42888	Support for flexible test suites
Submitted:	16 Feb 2009 15:49	Modified:	23 Mar 2009 17:58
Reporter:	Lars Heill	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tools: Pushbuild	Severity:	S4 (Feature request)
Version:	2	OS:	Any
Assigned to:	Daniel Fischer	CPU Architecture:	Any
Tags:	fr

Description:
Note, from serverPT AI 2009-02-04:
Daniel and Lars T to discuss with Kostja, Joro, Magnus, Alik, to ensure they understand/agree before the change is made.

Help make PB2 pushes green and easily manage flexible test suites:

Proposal was:

Add special make targets for test suites that PB2 pick up, and in particular one "experimental" suite that will never make the push look red in PB2 if tests fail.

Benefits:
– Easy for teams to configure which tests/suites run, which tests do not wave a red flag, no assistance from PB2/build team needed
– Devs can easily run tests same way as PB2
– Community can easily tell what tests we run internally
– Fully backwards compatible

From Lars T (under a "PB1" heading, probably misplaced?):

  We need a suite/marking for "experimental" tests.

  Requirements (as Lars sees it) for tests marked "experimental":
  - Tests shall *never* make the pushbuild boxes red or yellow,
  - Tests shall be executed in every pushbuild run,
  - Failure of test shall be clearly marked in pushbuild main page, 
    e.g. by a letter "E",
  - Failures debug info shall be easy to locate, e.g. by clicking
    on the "E" letter.

How to repeat:
N/A

Suggested fix:
From Daniel F in reply to Lars T:

>> 1. No test should ever fail with *red* in PB1/PB2 if the code is
>>    not broken, specifically:
>>
>>    a) it is not ok to have a test that sporadically shows red in
>>       pushbuild due to timing/load issues.  That tests needs to
>>       be re-written or disabled.
> 
> Agreed.
> 
> The colour red is currently used to indicate bugs in the server as well 
> as in test cases, and there isn't really a way to tell the difference. 
> Putting known-bad test cases into an experimental/unstable suite sounds 
> like a good way to mark test cases as failing randomly.
> 
>>    b) it is not ok to have a box show red because the *machine*
>>       is broken, out of disk, etc.  Those failures should be
>>       indicated with another color or some text.
> 
> I agree. It'd be nice if MTR could recognize this condition and use a 
> different status code for these failures.
> 
>> 2. A test case that needs to be fixed by a certain team should be
>>    enable-able for that team tree only.
>>
>>    (Example: rpl.rpl_sporadic_master that is sporadically failing
>>    should be enabled for rpl team trees only).
> 
> Agreed.
> 
>> 3. A test case that is sporadically failing, but that is very
>>    important for multiple teams needs to be enable-able for *all
>>    teams*, but should *not* show red in PB.  PB should instead
>>    print a text for this or use some other means of saying that
>>    the test failed.
> 
> Agreed.
> 
>> I think Daniel idea might solve this (I don't understand it
>> fully).  Also, I know that Magnus has been thinking about this.
> 
> My idea didn't address items 2 and 3 before. Here's a revised proposal:
> 
> 1) Someone creates a Makefile.test in mysql-test/.
> 2) Someone creates the target "test" that runs MTR with default
>    settings. Danny asks PB2 to use this make target if present instead
>    of determining which tests to run in some other way.
> 3) RPL team creates a target "test-rpl" that runs MTR with settings that
>    should be used for any -rpl* trees. Danny asks PB2 to look for a
>    make target test-XXX first when figuring out what tests to run in
>    a mysql-MAJOR.MINOR-XXX* branch.
> 3) Unstable tests are moved to an "unstable" test suite. Danny asks PB2
>    to not consider failures in any "unstable*" test suite a big deal.
> 4) RPL team creates an "unstable-rpl" test suite and adds it to the
>    "test-rpl" target. Unstable test cases that should only run in the
>    RPL branch are added here.
> 
> I could be the "someone" in steps 1 and 2 if this is accepted, since I 
> already know what this target should do in order for pb2 to realize what 
> is going on. :)
> 
> The main benefits that I see:
> - Teams can configure which tests run in their pb2 branches without
>   intervention from the build team
> - Teams can configure which tests cause red in their branches without
>   intervention from the build team
> - Developers can easily run tests in the same way as in pb2
> - The community can easily tell what kind of tests we run internally
>

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/67338

2739 Daniel Fischer	2009-02-24
      bug#42888: Add collections of test runs to make it both configurable and
      transparent what kinds of tests we run during integration testing.
      added:
        mysql-test/collections/
        mysql-test/collections/README
        mysql-test/collections/default.daily
        mysql-test/collections/default.push
        mysql-test/collections/default.weekly

Not patch pending yet, 5.1 and above need extra work for the experimental tests.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/67354

2793 Daniel Fischer	2009-02-24 [merge]
      Bug#42888 for MySQL 5.1. Add collections of test runs to make it both configurable and transparent what kinds of tests we run during integration testing. Implement filter list in mysql-test-run.pl to filter out failures of experimental test cases, using a new status code "exp-fail", so we can tell regressions ("fail") from failures of test cases that are still in development ("exp-fail").
      added:
        mysql-test/collections/
        mysql-test/collections/README
        mysql-test/collections/README.experimental
        mysql-test/collections/default.daily
        mysql-test/collections/default.experimental
        mysql-test/collections/default.push
        mysql-test/collections/default.weekly
      modified:
        mysql-test/lib/mtr_report.pm
        mysql-test/mysql-test-run.pl

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/67358

3035 Daniel Fischer	2009-02-24 [merge]
      Merge bug#42888 to 6.0, adjusting the list of default per-push test runs.
      added:
        mysql-test/collections/
        mysql-test/collections/README
        mysql-test/collections/README.experimental
        mysql-test/collections/default.daily
        mysql-test/collections/default.experimental
        mysql-test/collections/default.push
        mysql-test/collections/default.weekly
      modified:
        mysql-test/lib/mtr_report.pm
        mysql-test/mysql-test-run.pl

Bjørn, can you please review this, in particular the change to mtr? Nothing in the above patches should break anything that already exists.

I have a few comments:

mtr_report.pm:

 - You have an extra diff caused by adding spaces in 

   my $comment=      $tinfo->{'comment'};

   and the following lines.

 - In the loop testing if the test is experimental, a short comment
   explaining the parsing of * for wildcards would be nice :-)

mtr.pl:

 - I don't like putting an extra if() around all the file
   processing. I'd rather do

   open(FILE, "<", $opt_experimental)  # on one or two lines
     or mtr_error(...);

   <then process the file>

   This is also how it's done other places. Makes for easier reading
   IMHO.

- Again, a comment explaining the processing would be nice. It's not
  immediately obvious what s/( +|^)#.*$// does. ;-)

README.experimental:

+3) If any other line contains a black followed by a hash (#), the hash

"black"?  :-)

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/67554

3038 Daniel Fischer	2009-02-25 [merge]
      bug#42888 merge to 6.0
      modified:
        mysql-test/collections/README.experimental
        mysql-test/lib/mtr_report.pm
        mysql-test/mysql-test-run.pl

Patch looks fine now, OK to push

Pushed into 6.0.11-alpha (revid:bjorn.munch@sun.com-20090310133716-5qpotn2sggp621s1) (version source revid:bjorn.munch@sun.com-20090310133716-5qpotn2sggp621s1) (merge vers: 6.0.11-alpha) (pib:6)

Pushed into 5.1.33 (revid:bjorn.munch@sun.com-20090310134806-wnurk6dobx9u605b) (version source revid:bjorn.munch@sun.com-20090310134806-wnurk6dobx9u605b) (merge vers: 5.1.33) (pib:6)

Danny pushed this to 6.0-mtr and 5.1-mtr, later pushed to main branches

Pushed to 5.1 and 6.0 (main: 5.1.33 and 6.0.11-alpha)

Noted in 5.1.33, 6.0.11 changelogs.

mysql-test-run.pl now supports an --experimental=file_name option. It 
enables you to specify a file that contains a list of test cases that
should be displayed with the [ exp-fail ] code rather than [ fail ] 
if they fail.

README.experimental says:

<quote>

These files are to be used with the --experimental option of 
mysql-test-run.pl. Please look at its help screen for usage information.

</quote>

I am not able to find (in 6.0 main) any mention of --experimental on MTR's help screen (mtr --help), so it seems like this has been forgotten. Will you take care of this, or should I file a new bug report for it?

Please file a new bug for that.

I have updated open Bug #42988 about another option missing from --help, saying it should also cover other ones missing. So just add 100. :-)

Pushed into 5.1.34-ndb-6.2.18 (revid:jonas@mysql.com-20090508185236-p9b3as7qyauybefl) (version source revid:jonas@mysql.com-20090508100057-30ote4xggi4nq14v) (merge vers: 5.1.33-ndb-6.2.18) (pib:6)

Pushed into 5.1.34-ndb-6.3.25 (revid:jonas@mysql.com-20090509063138-1u3q3v09wnn2txyt) (version source revid:jonas@mysql.com-20090508175813-s6yele2z3oh6o99z) (merge vers: 5.1.33-ndb-6.3.25) (pib:6)

Pushed into 5.1.34-ndb-7.0.6 (revid:jonas@mysql.com-20090509154927-im9a7g846c6u1hzc) (version source revid:jonas@mysql.com-20090509073226-09bljakh9eppogec) (merge vers: 5.1.33-ndb-7.0.6) (pib:6)

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/73807

2730 Daniel Fischer	2009-05-12 [merge]
      merge bug#42888

Pushed into 5.1.36 (revid:joerg@mysql.com-20090611162132-wq3i5e6rqvcg0pgn) (version source revid:joerg@mysql.com-20090529134055-nnwgrfjdhr0ebmh8) (merge vers: 5.1.36) (pib:6)

Pushed into 5.0.84 (revid:joerg@mysql.com-20090611143247-ub720iread7cpftc) (version source revid:df@sun.com-20090512110916-k1bwfyyn006c8vb3) (merge vers: 5.0.83) (pib:6)

Pushed into 5.4.4-alpha (revid:anozdrin@bk-internal.mysql.com-20090710111017-bnh2cau84ug1hvei) (version source revid:mhansson@mysql.com-20090616095106-lunvwomidjv62ww3) (merge vers: 5.4.4-alpha) (pib:11)

Pushed into 5.1.37-ndb-7.0.8 (revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)

Pushed into 5.1.37-ndb-6.3.27 (revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (version source revid:jonas@mysql.com-20090826105955-bkj027t47gfbamnc) (merge vers: 5.1.37-ndb-6.3.27) (pib:11)

Pushed into 5.1.37-ndb-6.2.19 (revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (version source revid:jonas@mysql.com-20090825194404-37rtosk049t9koc4) (merge vers: 5.1.37-ndb-6.2.19) (pib:11)

Pushed into 5.1.35-ndb-7.1.0 (revid:magnus.blaudd@sun.com-20090827163030-6o3kk6r2oua159hr) (version source revid:jonas@mysql.com-20090826132541-yablppc59e3yb54l) (merge vers: 5.1.37-ndb-7.0.8) (pib:11)