MySQL Bugs: #37477: Order of error codes for --error statement in tests is significant

Bug #37477	Order of error codes for --error statement in tests is significant
Submitted:	18 Jun 2008 12:55	Modified:	8 Nov 2010 11:17
Reporter:	Øystein Grøvlen	Email Updates:
Status:	Won't fix	Impact on me:	None
Category:	Tools: MTR / mysql-test-run	Severity:	S3 (Non-critical)
Version:		OS:	Any
Assigned to:	Bjørn Munch	CPU Architecture:	Any

Description:
1) If specifying in a test that a statement should either succeed or fail with a given error, 0 needs to be the first error code.  If another error code is listed first, test will fail if the statement execute successfully.

2) As above, but statement fails with expected error code.  When error code (not 0) is listed first "Got one of the listed errors" is printed.  If 0 is listed first, this is not written 

How to repeat:
1)
Fails when SELECT succeeds:
--error 1054,0
SELECT a FROM t;

Does not fail when SELECT succeeds:
--error 0,1054
SELECT a FROM t;

2)
If b is not a valid column, this prints "Got one of the listed errors":
--error 1054,0
SELECT b FROM t;

This prints nothing:
--error 0,1054
SELECT b FROM t;

This is actually documented behavior, the reason 0 has to be first is that it has a different semantics from any other number: it means the statement does not fail. A 0 elsewhere in the list would be interpreted as the statement failing with error code 0.

It's possible to allow 0 with this special meaning also later in the list, but I'm not sure this is a good idea: it would lead people to write tests which then cannot be backported but will give unexpected failures if run on versions without this "fix".

As explained, this is documented and easily worked-around behavior, and will not be fixed in order to maintain backwards compatibility for tests.