MySQL Bugs: #53535: Generated MTR tests not runnable when MTR command follows comment

Bug #53535	Generated MTR tests not runnable when MTR command follows comment
Submitted:	10 May 2010 9:33	Modified:	13 Oct 2010 13:34
Reporter:	John Embretsen	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tools: Random Query Generator	Severity:	S3 (Non-critical)
Version:		OS:	Any
Assigned to:	Bernt Marius Johnsen	CPU Architecture:	Any

Description:
When the RQG is set up to generate MTR tests based on results from certain groups of SQL statements, such generated test files should be runnable in MTR without modification.

Some generated tests fail if an SQL-style comment (e.g. /* This is a comment */) is immediately followed by an MTR command (e.g. --disable_abort_on_error). Details:

Currently, a generated test file may start with something like this:

----- start quote ------

/* Server0: MySQL 6.0.14-alpha-debug-log */

--disable_abort_on_error
SET SESSION optimizer_switch = 'index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off,mrr=on,mrr_cost_based=off,index_condition_pushdown=on';
SET SESSION optimizer_join_cache_level = 8;
SET SESSION debug = '';
--enable_abort_on_error

/* Begin test case for query 0 */

--disable_warnings
DROP TABLE /*! IF EXISTS */ E;
--enable_warnings

----- end quote ------

MTR interprets the first three lines as a single SQL statement and sends it to the server, since the first comment within "/* */" is not terminated (with a semicolon):

-----

mysqltest: At line 1: query '/* Server0: MySQL 6.0.14-alpha-debug-log */
--disable_abort_on_error
SET SESSION optimizer_switch = 'index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,firstmatch=on,loosescan=on,materialization=on,semijoin=off,mrr=on,mrr_cost_based=off,index_condition_pushdown=on'' failed: 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '--disable_abort_on_error
SET SESSION optimizer_switch = 'index_merge=on,index_me' at line 2
Failed to read from (...), errno: 22

-----

Adding a semicolon at the end of the lines containing SQL comments only makes MTR interpret the input correctly.

Observed against mysql-6.0-codebase-bugfixing as of 2010-05-05.

How to repeat:
Run a RQG command which generates MTR test cases upon "failure", e.g.:

perl ./runall.pl \
--threads=1 \
--grammar=conf/optimizer/outer_join.yy \
--queries=10000 \
--basedir=$CODE \
--vardir=$PWD/var-transformer \
--validator=Transformer \
--views

This results in the RQG saying where to find the generated test (also note bug#53532):

# 2010-05-10T10:37:49 MySQL test dumped to /tmp//1273480669.test

Run the generated tests in MTR:

cd /path/to/mysql-test
cp  /tmp//1273480669.test t/mytest.test
perl mysql-test-run.pl mytest --record

Suggested fix:
Terminate each RQG-generated comment-only statements/lines in MTR tests with a semicolon?

Similarly, for comments at the end of a test such as:

/* End of test case for query 1 */

MTR fails with:

mysqltest: At line 309: End of line junk detected: "/* End of test case for query 1 */
"

Need a new "How to repeat" since the given example does not repeat the problem. A fix is ready, but need to be verified.

Possible fix

Attachment: bug_53535.patch (application/x-subversion-stat, text), 3.10 KiB.

The attached patch (bug_53535.patch) has one if-test reversed and a typo:

Reversed if-test, causing the opposite behavior than desired:

    if ($useHash) {
        return "/* ". $splitLine . " */"; 
    } else {
        return "# " . $splitLine;
    }

 which should be

    if ($useHash) {
        return "# " . $splitLine;
    } else {
        return "/* ". $splitLine . " */"; 
    }

There is also a typo in the following comment (harmless):

    ## We use Hash-comments in an pure MySQL environment due to MTR
    ## limtations

 Change "limtations" to "limitations".

With these changes the patch seems to work fine, at least with the simple scenarios I have tested.

One way of testing a patch for this issue is to do the following:

1) Prepare one version of MySQL and refer to its basedir as $CODE
2) Prepare another version of MySQL and refer to its basedir as $CODE2
3) Save a grammar file "mygrammar.yy" with the following contents:

  query_init:
    # Insert the "minor" version digit into table B. 
    # Will cause diffs in comparison testing between minor releases.
    INSERT INTO B (`col_varchar_nokey`) ( SELECT SUBSTR(@@version FROM 3 FOR 1) );

  query:
    SELECT pk, `col_varchar_nokey` FROM B;

4) Run the following RQG command from the randgen tree:

 perl ./runall.pl \
  --basedir1=$CODE \
  --basedir2=$CODE2 \
  --vardir1=$PWD/var-compare/vardir-1 \
  --vardir2=$PWD/var-compare/vardir-2 \
  --threads=1 \
  --queries=2 \
  --grammar=$PWD/mygrammar.yy \
  --validator=ResultsetComparatorSimplify \
  --reporters=Shutdown \
  --duration=30 \
  --engine=InnoDB

You should see a diff and a reference to a generated simplified .test file as long as the two MySQL versions differ in their "minor version" digit.