Bug #37477 Order of error codes for --error statement in tests is significant
Submitted: 18 Jun 2008 12:55 Modified: 8 Nov 2010 11:17
Reporter: Øystein Grøvlen Email Updates:
Status: Won't fix Impact on me:
None 
Category:Tools: MTR / mysql-test-run Severity:S3 (Non-critical)
Version: OS:Any
Assigned to: Bjørn Munch CPU Architecture:Any
Triage: D4 (Minor)

[18 Jun 2008 12:55] Øystein Grøvlen
Description:
1) If specifying in a test that a statement should either succeed or fail with a given error, 0 needs to be the first error code.  If another error code is listed first, test will fail if the statement execute successfully.

2) As above, but statement fails with expected error code.  When error code (not 0) is listed first "Got one of the listed errors" is printed.  If 0 is listed first, this is not written 

How to repeat:
1)
Fails when SELECT succeeds:
--error 1054,0
SELECT a FROM t;

Does not fail when SELECT succeeds:
--error 0,1054
SELECT a FROM t;

2)
If b is not a valid column, this prints "Got one of the listed errors":
--error 1054,0
SELECT b FROM t;

This prints nothing:
--error 0,1054
SELECT b FROM t;
[2 Mar 2010 13:47] Bjørn Munch
This is actually documented behavior, the reason 0 has to be first is that it has a different semantics from any other number: it means the statement does not fail. A 0 elsewhere in the list would be interpreted as the statement failing with error code 0.

It's possible to allow 0 with this special meaning also later in the list, but I'm not sure this is a good idea: it would lead people to write tests which then cannot be backported but will give unexpected failures if run on versions without this "fix".
[8 Nov 2010 11:17] Bjørn Munch
As explained, this is documented and easily worked-around behavior, and will not be fixed in order to maintain backwards compatibility for tests.