Bug #53949 Allow MTR2 to start server without hardcoded command-line options
Submitted: 24 May 2010 20:44 Modified: 14 Oct 2010 15:33
Reporter: Elena Stepanova Email Updates:
Status: Closed Impact on me:
None 
Category:Tools: MTR / mysql-test-run Severity:S4 (Feature request)
Version:Any OS:Any
Assigned to: Bjørn Munch CPU Architecture:Any

[24 May 2010 20:44] Elena Stepanova
Description:
Currently if we start a server (or run a test without specific configuration requirements), MTR2 not only generates a config file with a number of options different from server defaults, but also adds some options to the command line.

In some tests, for example SystemQA tests, we use MTR for starting server(s) via --start option (or, we'll switch to --start-and-exit as soon as fix for bug#51767 is merged into main trees), and then run tests bypassing MTR. Default MTR configuration is not suitable for the tests, so we want to generate our own configuration files and pass it to MTR via --defaults-file option. However, now it is not a solution, as we still need to override command-line options which MTR adds unconditionally. Thus, we have to maintain separate sets of options -- one set for config file and another set for command line, which makes test framework structure cumbersome (and unreliable, too, as we do not have full control over server configuration).

What we need is ability to tell MTR stick strictly to requested configuration without adding/changing anything on its own. For example, if we run MTR as

perl ./mysql-test-run.pl  --start --defaults-file=./my1.cnf --mysqld=--innodb_lock_wait_timeout=10

we expect to have server command line look like
mysqld --defaults-file=<cnf file> --innodb_lock_wait_timeout=10
where <cnf file> is an exact copy of my1.cnf (or path to the original file); while currently the command line would contain few more options.

One exception is --defaults-group-suffix which MTR also adds to the command line; if it is strictly needed due to current MTR design, it is acceptable as we can always generate config file with the corresponding sections.

How to repeat:
run 
perl ./mysql-test.run.pl --start

observe the command line in processlist

Suggested fix:
If there is a reason why some options are hardcoded rather than written into configuration files when needed, the fix might be as simple as adding yet another MTR option to skip them. This option only would control options which MTR adds voluntarily, while extra server options given explicitely in MTR command line should obviously be preserved.
[8 Jun 2010 8:00] Bjørn Munch
There is a reason why some options are given on the mysqld command line and not in the cfg file. The cfg is generated once for each mtr run and is valid for all tests, while command line options are used for whatever varies between tests within the test run, e.g. innodb vs. non-innodb tests. We can't simply remove these.

Are there any *particular* command line options you need to override/remove?
[8 Jun 2010 8:49] Elena Stepanova
As said in the description, I'm questioning not the way MTR runs any particular sets of tests, but the way it handles "testless" --start mode. When I do "perl mysql-test-run.pl --start", there is no room for second-guessing what could vary between test runs as there are no test runs, hence no need to add any options that were not requested. However, I am still getting

mysqld --defaults-group-suffix=.1 --defaults-file=var/my.cnf --log-output=file --loose-skip-innodb --loose-skip-log-bin --core-file --loose-debug-sync-timeout=300

Ideally I would like to have a way to get rid of *all* options above and have only those which I pass to MTR explicitly, be it --defaults-file (or --no-defaults, for that matter), or any mysqld command-line options. 

These two are a big problem as they have to be overridden regularly:
--loose-skip-innodb
--loose-skip-log-bin

This is not that bad, but still when I need to change it, lets say switch off logging for stress testing, or turn on TABLE logging for some specific purposes, it's a trouble.
--log-output=<whatever>

--defaults-group-suffix=.1 is also a bit of pain when I want to use a side configuration file, but it is not that critical.

--core-file 
--loose-debug-sync-timeout=300
are usually all right, but again -- if I want to change them on some reason, it's a fuss.

More importantly still, when I use MTR for starting server, I need to ensure that no other options will silently appear with time.

If there is a possibility that the change in behavior could affect any existing tests or other MTR usage, as also said before, I would be happy with a non-default MTR option that switches on the "clean" start as described above.
[8 Jun 2010 8:57] Bjørn Munch
Right, I see the point. The reason --start does it like that even though it's not planning to run many tests, is simply because the method for starting the server is common for all cases, be it many tests, a single test, or --start without actually running a test. There has been no need to treat these differently, so they share the implementation of server start.

Perhaps a solution could be for --start or --start-and-exit to change the logic, and put the options into the cfg file instead of the command line. This adds some complexity but could hopefully solve your problem?
[8 Jun 2010 9:04] Elena Stepanova
Yes it would be fine as long as MTR does not make any changes in my configuration file if I pass it via --defaults-file option.

That is,

if I run 
MTR --start <mysqld options of my choice>
MTR does whatever it wants with its config file, and starts server as
mysqld --defaults-file=<config file generated by MTR> <mysqld options of my choice>

and if I run
MTR --start --defaults-file=<my config file> <any mysqld options of my choice>
MTR does *nothing* with my config file, and starts server as
mysqld --defaults-file=<my config file> <mysqld options of my choice>
[8 Jun 2010 12:23] Bjørn Munch
OK, looks like what you need is for MTR not to make its own config file *nor* add any options. That sounds doable. There should be an mtr option for this, and it would require --defaults-file and accept (but not require) additional --mysqld options.
[9 Jun 2010 8:24] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110576

2911 Bjorn Munch	2010-06-09
      Bug #53949 Allow MTR2 to start server without hardcoded command-line options
      Added option --user-args, to be used with --start*
      Only keeps --defaults-file and --defaults-group-suffix
      Also added missing help text entry for --start-and-exit
[15 Jun 2010 8:17] Magnus Blåudd
Just wondering why the extrenal test framework can't start the mysqld  by itself? Well, well... approved anyway.
[15 Jun 2010 11:10] Bjørn Munch
Pushed to 5.1-mtr, trunk-mtr, next-mr-mtr
[16 Aug 2010 6:30] Bugs System
Pushed into mysql-trunk 5.6.1-m4 (revid:alik@sun.com-20100816062701-qo9dpnk5tkt1pksb) (version source revid:alik@sun.com-20100816062603-xc16eftmv7rmktyq) (merge vers: 5.6.1-m4) (pib:20)
[16 Aug 2010 6:34] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100816062819-bluwgdq8q4xysmlg) (version source revid:alik@sun.com-20100816062612-enatdwnv809iw3s9) (pib:20)
[16 Aug 2010 17:06] Paul DuBois
Test suite changes. No changelog entry needed.
[28 Sep 2010 8:47] Bugs System
Pushed into mysql-5.1 5.1.52 (revid:sunanda.menon@sun.com-20100928083322-wangbv97uobu7g66) (version source revid:sunanda.menon@sun.com-20100928083322-wangbv97uobu7g66) (merge vers: 5.1.52) (pib:21)
[14 Oct 2010 8:29] Bugs System
Pushed into mysql-5.1-telco-7.0 5.1.51-ndb-7.0.20 (revid:martin.skold@mysql.com-20101014082627-jrmy9xbfbtrebw3c) (version source revid:martin.skold@mysql.com-20101014082627-jrmy9xbfbtrebw3c) (merge vers: 5.1.51-ndb-7.0.20) (pib:21)
[14 Oct 2010 8:43] Bugs System
Pushed into mysql-5.1-telco-6.3 5.1.51-ndb-6.3.39 (revid:martin.skold@mysql.com-20101014083757-5qo48b86d69zjvzj) (version source revid:martin.skold@mysql.com-20101014083757-5qo48b86d69zjvzj) (merge vers: 5.1.51-ndb-6.3.39) (pib:21)
[14 Oct 2010 8:58] Bugs System
Pushed into mysql-5.1-telco-6.2 5.1.51-ndb-6.2.19 (revid:martin.skold@mysql.com-20101014084420-y54ecj85j5we27oa) (version source revid:martin.skold@mysql.com-20101014084420-y54ecj85j5we27oa) (merge vers: 5.1.51-ndb-6.2.19) (pib:21)
[14 Oct 2010 15:33] Jon Stephens
Tesing only; closed.