MySQL Bugs: #36597: Testsuite "jp": Suspicious results for some tests using sjis

Bug #36597	Testsuite "jp": Suspicious results for some tests using sjis
Submitted:	8 May 2008 15:17	Modified:	16 Sep 2008 4:50
Reporter:	Matthias Leich	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Charsets	Severity:	S3 (Non-critical)
Version:	5.1	OS:	Any
Assigned to:	Alexander Barkov	CPU Architecture:	Any

Description:
The following tests of the "jp" testsuite give suspicious
results differing from the old expected ones:
---------------------------------------------------------
t/jp_charset_sjis.test:
Warning during load std_data/jisx0208_sjis.dat

t/jp_charset_ujis.test:
Warning during load std_data/jisx0208_ujis.dat

t/jp_charset_utf8.test
Warning during load std_data/jisx0208_utf8.dat

t/jp_convert_sjis.test:
Warning during load std_data/jisx0208_sjis.dat + result sets

t/jp_convert_ujis.test:
Warning during load std_data/jisx0208_ujis.dat

t/jp_convert_utf8.test:
Warning during load std_data/jisx0208_utf8.dat

t/jp_like_sjis.test:
Warning during load std_data/jisx0208_sjis.dat

t/jp_like_ujis.test:
Warning during load std_data/jisx0208_ujis.dat

t/jp_like_utf8.test:
Warning during load std_data/jisx0208_utf8.dat

t/jp_select_sjis.test:
Warning during load std_data/jisx0208_sjis.dat + result sets

t/jp_select_ujis.test:
Warning during load std_data/jisx0208_ujis.dat

t/jp_select_utf8.test:
Warning during load std_data/jisx0208_utf8.dat

t/jp_where_ujis.test:
Warning during load std_data/jisx0208_ujis.dat

t/jp_where_utf8.test:
Warning during load std_data/jisx0208_utf8.dat

Some notes:
-----------
1. The development of the "jp" testsuite ended
   around beginning of 2005.
   The author of this suite is no more within
   the company since that year.
2. AFAIR the state in 2005 was that most or all 
   tests passed. It is at least not very probable
   that the warnings around load data from some
   infiles occured at that time.
3. The activities around
   Bug#28563 Test suite "jp" fails completely
   showed that
   - the development of this suite was partially
     not completed
   - the server and the testing tools have 
     significant changed (Example: handling of
     storage engines, joins with USING) since
     beginning of 2005.
     This caused that the suite was no more
     runnable.
   This leads to the conclusion that the "jp"
   suite was simply ignored since a long time.
4. Most tests where warnings during load data
   occur do NOT suffer from differences between
   expected and current result sets.
5. I did not found open bugs which seem to match
   the current issues with the "jp" suite.

I am unable to decide where the current bug
belongs to. There could be an unknown server
bug or the affected tests are the "victim" of
server improvements and have to be adjusted
to the current behaviour.

My environment:
5.1-bugteam last ChangeSet@1.2606, 2008-05-05
5.0-bugteam ChangeSet@1.2622, 2008-05-06
    The 5.0 trees do not contain the "jp" suite.
    I copied the tests including the fix 
    for Bug#28563 into this tree and observed
    the same problems.
- BUILD/compile-pentium-debug-max
- Intel Core2Duo (64 Bit)
- OpenSuSE 10.3 (64 Bit)

How to repeat:
Please wait till
  Bug#28563 Test suite "jp" fails completely
is fixed and run after that
./mysql-test-run.pl --suite=jp --force

I rerun the tests with
mysql-5.1-bugteam last ChangeSet@1.2639, 2008-05-15
and all warnings during LOAD DATA disappeared.

All queries with result set differences share the
following properties:
1. They work on a table which was filled via
   LOAD DATA LOCAL
   INFILE 'suite/jp/std_data/jisx0208_sjis2.dat' ...
   All other INFILES are harmless.
2. Number of recorded result set lines
   old: 1 new: 2
   An additional SELECT COUNT(*) shows that we 
   have now two result rows and not just an 
   additional line break within column values.
   The difference between recorded old and new
   result set is only this additional row with
   column values = empty string. 
3. There is only one line within the INFILE.
   It starts with (copied from my editor, I 
   guess the font used does not contain the 
   corresponding japanese characters)
   <83>\<8f>\<95>\<97>\<8d>\<89>\<8e>\<90>
   The line being in the recorded old and new
   result sets starts with
   <83>\<8f>\<95>\<97>\<8d>\<89>\<8e>\<90>
Experiments:
1. Add a line with 'A' at the beginning of the infile
   --> We get an additional result set line
       with 'A', but the suspicious line with 
       values = empty string does not disappear.
2. Modify the line within the infile to
   'A<83>\<8f>\<95>\<97>\....'
   --> Protocol line is now  'A<83>\<8f>\<95>\<97>\...'
       but the suspicious line with values = empty
       string does not disappear.

So it looks like LOAD DATA misinterprets now the data
within the INFILE and generates the suspicious row.

How to reproduce the bug:
-------------------------
The 'jp' testsuite with fixes for
  Bug#28563 Test suite "jp" fails completely
is since 2008-05-19 in 5.1.25-rc.

./mysql-test-run.pl --enable-disabled --skip-ndb \
   --testcase-timeout=20 --suite=jp jp_convert_sjis
or
./mysql-test-run.pl --enable-disabled --skip-ndb \
   --testcase-timeout=20 --suite=jp jp_select_sjis

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/50623

2681 Alexander Barkov	2008-07-29
      Bug#36597 Testsuite "jp": Suspicious results for some tests using sjis
      Problem: Extra new line appeared in jisx0208_sjis2.dat in a mistake, which
      broke jp_convert_sjis and jp_select_sjis tests.
      Fix: removing extra line
      
        mysql-test/suite/jp/std_data/jisx0208_sjis2.dat:
            removing wrong extra new line
      
        mysql-test/suite/jp/t/disabled.def:     
            activating disabled tests

Pushed into 5.1.28-bugteam and 6.0.7-bugteam.

Pushed into 5.1.28  (revid:bar@mysql.com-20080729054357-nsz5lb0blzvz0w6h) (version source revid:davi.arnaut@sun.com-20080826183817-r22ie0hgagpcn6su) (pib:3)

Test case changes. No changelog entry needed.

Setting report to NDI pending push into 6.0.x.

Pushed into 6.0.7-alpha  (revid:bar@mysql.com-20080729054357-nsz5lb0blzvz0w6h) (version source revid:hakan@mysql.com-20080725175322-8wgujj5xuzrjz3ke) (pib:3)

No changelog entry needed.