MySQL Bugs: #55264: Excluding tests from Windows does not work reliably

Bug #55264	Excluding tests from Windows does not work reliably
Submitted:	14 Jul 2010 15:22	Modified:	10 Jan 2013 11:31
Reporter:	Joerg Bruehe	Email Updates:
Status:	Can't repeat	Impact on me:	None
Category:	Tests: Server	Severity:	S7 (Test Cases)
Version:	5.1.45, 7.1.5	OS:	Windows
Assigned to:	Assigned Account	CPU Architecture:	Any

Description:
In the build of Cluster-7.1.5, I detected a new occurrence of bug#49984.
That bug is reported to be fixed in 5.1.45, my build is based on 5.1.47, the fix is included (checked), but still the test is taken on Windows and fails.

The fix was to exclude it by adding "--source include/not_windows.inc", see http://lists.mysql.com/commits/99371

Checking server build logs of 5.1.*, I see that the test is still taken on Windows even in 5.1.45 and up, in spite of this "--source" statement.
However, "rpl_cross_version" uses the same "--source", and it does get skipped on Windows as intended.

How to repeat:
Run the "rpl" suite on Windows and notice that "rpl_loaddata_symlink" is taken even though it should be skipped.

The problem isn't so much that we have a failing test,
it is that a mechanism does not work reliably as intended.
That's why I consider this one more severe than usual.

As I wrote in the report: Also seen in general 5.1 server builds.

Specifically, it occurs in 5.1.42, 5.1.43, 5.1.44, 5.1.45, 5.1.47, 5.1.48, and 5.1.49.

In 5.1.40, 5.1.41, and 5.1.46, there were general issues with the test suite, so the test was never attempted (there are no "skipped" messaged logged).

I have not checked versions before 5.1.40.

I was not able to reproduce this bug with 5.1 (mysql-5.1.50-win32.msi and the development version).
Reason:
- The mysql-test-run.pl script (see "mtr_cases.pm", line 828) rejected the execution of the test "rpl_loaddata_symlink.test" with "No tests with sh scripts on Windows". This always happens if a "...-master.sh" is existing to a test, which is a shell script not executable on Windows.

This bug still occurs.

It is still affecting test "rpl_loaddata_symlink",
and it happens in the general server (5.1.51, 5.1.52 and earlier ones)
as well as in Cluster (7.1.9).

According to the test log, "version_compile_os" is set to "Win32" or "Win64" as expected, so I can't tell why this doesn't work as expected.

Can't repeat:

"I run main and separately rpl tests on win7 and all test were successful:
 
Completed: All 1092 tests were successful."