MySQL Bugs: #29552: Please decouple test logic from Makefile

Bug #29552	Please decouple test logic from Makefile
Submitted:	4 Jul 2007 14:38	Modified:	8 Jun 2009 15:55
Reporter:	Ingo Strüwing	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tools: MTR / mysql-test-run	Severity:	S4 (Feature request)
Version:	Any	OS:	Any
Assigned to:		CPU Architecture:	Any

Description:
Makefile contains test logic in the sense that it determines which tests to run when called as "make test" or "make test-force".

The problem is that even "make test-force" stops after the first test sequence (e.g. normal test mode with statement-based replication) if some problem was detected (e.g. non-empty warnings file). Hence the other sequence(s) are not run. Many problems may hide in the not run later sequence(s). This is difficult to fix in a makefile.

Another problem is that it is not possible to specify options like --testcase-timeout=30 --suite-timeout=240 with "make test". On my systems with not so recent hardware the default timeouts are slightly too low. I am forced to run mysql-test-run.pl directly, risking to forget a sequence when one is added to Makefile.

How to repeat:
Run "make test-force" on an older system. See timeouts to happen. Notice that testing stops after the first sequence that reported some failure or warnings.

Suggested fix:
Create a shell/perl script mysql-test/mtr-default (or a similar name) that runs the sequences in the standard order. Make it able to pass options (and ideally test names) to mysql-test-run.pl.

Alternatively intergrate this logic into mysql-test-run.pl and activate it with an option (e.g. --default-tests and --default-tests-force).

This can then be called by "make test" and "make test-force".

Thank you for a reasonable feature request.

there's no need for a special script.
If you want 'make test-force' to continue after the first failed test sequence, you can start it as 'make -k test-force' or add '-' (or '-k') to the Makefile, to make the change permanent.

Test parameters could be passed to the make, see for example how test-force-full target works.

will make -k return a non-zero return code if the first sequence failed, but the second one suceeded? Portably?

Outdated.