Bug #52019 main.mysqltest fails on new tests for lowercase_result
Submitted: 12 Mar 2010 21:07 Modified: 14 Nov 2010 2:52
Reporter: Karen Langford Email Updates:
Status: Closed Impact on me:
Category:Tools: MTR / mysql-test-run Severity:S3 (Non-critical)
Version:5.1.45 OS:IBM AIX
Assigned to: Bjørn Munch CPU Architecture:Any
Tags: test_failure

[12 Mar 2010 21:07] Karen Langford
This bug has only just appeared in 5.1.45, occurring on aix5.3 powerpc 32/64
bit platforms. Test seems to be having a problem with the 'test for lowercase_result' tests added to mysqltest: possibly test 7? (just guessing,
its the next test but the results don't appear)

# 7. Test missing lower casing of "unknown" characters
--character_set utf8
SELECT 0 as "WILL NOT lower case \304\313\320";
--character_set latin1

main.mysqltest                           [ retry-fail ]  timeout after 900 seconds
        Test ended at YYYY-MM-DD HH:MM:SS

Test case timeout after 900 seconds

== /PATH/mysql-test/var/log/mysqltest.log ==
select "500g bl\345b\346rsyltet\370y" as "will be lower cased";
will be lower cased
500g bl\345b\346rsyltet\370y
SELECT "UPPER" AS "WILL NOT BE lower cased";
WILL NOT BE lower cased
select "abcdef" as "uvwxyz";
select "xyz" as name union select "abc" as name order by name desc;
select 1 as "some new text";
some new text

 == /PATH/mysql-test/var/tmp/analyze-timeout-mysqld.1.err ==
mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket
 '/data0/mysqldev/tmp-201003011009-5.1.45-6768/aix5.3-powerpc-64bit/test/mysql-test/var/tmp/mysqld.1.s' (79)

How to repeat:
Found when running the full test suite.
[25 Mar 2010 18:31] [ name withheld ]
I observe this test hanging on Fedora 12 PPC/PPC64 as well; confirm the misbehavior is new as of .45.
[26 Mar 2010 1:50] [ name withheld ]
I've looked into this problem and identified multiple related bugs in mysqltest.cc.
The direct cause of the infinite loop is that this code is wrong:

	  /* It was not a multiline char, push back the characters */
	  /* We leave first 'c', i.e. pretend it was a normal char */
	  while (p > mb_start)

Contrary to what the comment claims, the loop pushes back all bytes of
a multi-byte character; so the next iteration of the outer loop of read_line()
starts at the same place, does the same thing again, lather rinse repeat
forever.  Fix the loop test so that it actually does leave the first byte
behind, and the infinite loop goes away.

Having seen this, the question is not why it fails on PPC, but how come
it manages to avoid failing everywhere.  Further investigation showed
that the culprit is the preceding call to my_mbcharlen(), which is passing
a "char" to an argument that is "uint".  On PPC, where char is unsigned
by default, this behaves as intended and you get a charlen that is > 1,
and then if the multibyte character isn't actually validly coded (ie,
exactly the test case added in main.mysqltest in 5.1.45), hello infinite loop.
But if char is signed by default, then something like \304 promotes to a
negative integer and then to a very large unsigned integer, causing
my_mbcharlen_utf8() to punt and return charlen = 0.  So on signed-char
machines the intended skipping of additional bytes of a multibyte char
fails to happen.  In most encodings this is fairly harmless since the
outer loop won't recognize those bytes as anything in particular, but there
are probably cases where bad things would happen.

While I was sniffing around to find these problems, I noticed a few other
bugs.  The feof() test inside the loop that accumulates extra bytes of a
multibyte character ought to be after the my_getc call not before it.  This is
relatively unlikely to be a problem in practice, but it's still wrong; and
someone might copy the erroneous coding pattern to somewhere else where it's
more likely to cause real problems.  Also, in replace_dynstr_append_mem(), the
temporary array lower[] is being declared in an inner block but the code
relies on it remaning valid until the end of the function.  You need to
declare it at function scope instead to ensure the compiler doesn't optimize
it out from under you.  Since that code is brand new in 5.1.45, I don't think
you should assume the lack of problem reports so far proves its portable.

The attached patch covers the above problems.  There were also two things
I noticed but didn't bother to change.  On a machine where char is signed,
my_getc will return different values for the original return of a high-bit-set
character (eg, \304) versus when it is returned again after being pushed back
through line_buffer[] (then you'll get a sign-extended version).  This seems
to be harmless for now because most callers just store the value right back
into a "char" variable, but it's going to bite you on the rear someday.  I'd
suggest a cast, or maybe change line_buffer to be unsigned char[] altogether.
Another thing that's dangerous with signed chars is that match_delimiter() is
comparing ints to chars.  I think it will accidentally fail to malfunction as
long as delimiter strings never contain any high-bit-set characters, but if
they ever do, look out.  (Oh, and you'd better think about the case of hitting
EOF while trying to match a multibyte delimiter string, too.)

I wonder BTW whether this particular call of my_mbcharlen() is the only one
with this sort of bug.  Perhaps you ought to consider changing its argument
type to char, instead of expecting every caller to be careful about casting.
[26 Mar 2010 1:52] [ name withheld ]
patch for some of the problems identified here

Attachment: mysql-lowercase-bug.patch (application/octet-stream, text), 1.59 KiB.

[11 Oct 2010 7:58] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:


2940 Bjorn Munch	2010-10-11
      Bug #52019 main.mysqltest fails on new tests for lowercase_result
      Fixed as suggested above
      Confirmed on Linux PPC and on AIX 5.3
[20 Oct 2010 9:24] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:


2943 Bjorn Munch	2010-10-20
      Bug #52019 main.mysqltest fails on new tests for lowercase_result
      Limited to actual bug fix, fixing a while condition
      Again confirmed on Linux PPC and on AIX 5.3
[20 Oct 2010 10:24] Bjørn Munch
Pushed to -mtr branches. I also fixed each of the separate identified problems with separate commits.
[29 Oct 2010 15:35] Paul DuBois
Changes to test suite. No changelog entry needed.
[13 Nov 2010 16:22] Bugs System
Pushed into mysql-trunk 5.6.99-m5 (revid:alexander.nozdrin@oracle.com-20101113155825-czmva9kg4n31anmu) (version source revid:alexander.nozdrin@oracle.com-20101113152450-2zzcm50e7i4j35v7) (merge vers: 5.6.1-m4) (pib:21)
[13 Nov 2010 16:29] Bugs System
Pushed into mysql-next-mr (revid:alexander.nozdrin@oracle.com-20101113160336-atmtmfb3mzm4pz4i) (version source revid:vasil.dimov@oracle.com-20100629074804-359l9m9gniauxr94) (pib:21)
[18 Nov 2010 15:55] Bugs System
Pushed into mysql-5.1 5.1.54 (revid:build@mysql.com-20101118153531-693taxtxyxpt037i) (version source revid:build@mysql.com-20101118153531-693taxtxyxpt037i) (merge vers: 5.1.54) (pib:21)
[16 Dec 2010 22:27] Bugs System
Pushed into mysql-5.5 5.5.9 (revid:jonathan.perkin@oracle.com-20101216101358-fyzr1epq95a3yett) (version source revid:jonathan.perkin@oracle.com-20101216101358-fyzr1epq95a3yett) (merge vers: 5.5.9) (pib:24)