MySQL Bugs: #47141: "mysqltest" breaks because it tries to act on a debug option which is disabled

Bug #47141	"mysqltest" breaks because it tries to act on a debug option which is disabled
Submitted:	4 Sep 2009 13:59	Modified:	9 Feb 2011 19:22
Reporter:	Joerg Bruehe	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tools: MTR / mysql-test-run	Severity:	S3 (Non-critical)
Version:	5.1.40, 5.4.2-beta	OS:	Windows
Assigned to:	Bjørn Munch	CPU Architecture:	Any

Description:
During the build of 5.4.2-beta, I had this happen on several tests.
Some of these also failed for other reasons in other runs,
but for some this issue is their only failure.

Example of this "specific" category:

=====
binlog.binlog_incident 'row'             [ fail ]
        Test ended at YYYY-MM-DD HH:MM:SS

CURRENT_TEST: binlog.binlog_incident
g:/mysql-5.4.2-beta-winbuild/mysql-5.4.2-beta-winx64/bin/mysqltest.exe: ERROR: Option 'debug=d:t:A,/mysql-5.4.
2-beta-winbuild/mysql-5.4.2-beta-winx64/mysql-test/var/log/mysqltest.trace' used, but is disabled

 - saving '/PATH/mysql-test/var/log/binlog.binlog_incident-row/' to '/PATH/mysql-test/var/log/binlog.binlog_in
cident-row/'
The servers were restarted 10 times
Spent 0.000 of 83 seconds executing testcases

Failed 10/10 tests, 0.00% were successful.
=====
The whole test run then ends, obvbiously, because the limit of 10 failures has been reached.

This happens for both the 32 bit and the 64 bit build on Windows.

The list of test cases:

binlog.binlog_incident
binlog.binlog_row_innodb_stat
binlog.binlog_row_mix_innodb_myisam
binlog.binlog_stm_innodb_stat
binlog.binlog_stm_mix_innodb_myisam
binlog.binlog_stm_unsafe_warning
main.fulltext_plugin
main.plugin
main.plugin_load
main.udf

How to repeat:
Run one of these tests with "debug" Windows binaries.

Suggested fix:
Probably, it is some build or environment setting, but I don't know which.

Also happened in the build of 5.1.40.

I can reproduce this with mtr --debug, but only if running with a debug mysqld binary but NON-debug mysqltest. Is there something strange in your environment? You might see this if you have built both with and without debug in the same sandbox.

I won't judge whether the Windows build environment qualifies as containing something "strange".  ;)

I will have to check the Windows build scripts for details, but I can easily imagine that we use the "mysqltest" of an optimized build to test a debug server there.

On Unix, this does not happen, because we do a full debug build for testing, just throw it all away except the server (which is renamed and then included in the binary package).

So your analysis is that an optimized "mysqltest" may show these reactions if asked to test a debug server?

To last question in previous comment: yes, if run with the --debug option, mtr will add the debug=... to both mysqltest and mysqld IF mysqld has been compiled with debug, but a non-debug mysqltest does not support it.

A workaround could possibly be to change the ordering of the directories in which mtr looks for the msqltest executable; at least this could cover the case where both debug and non-debug are available. Or we could simply skip the non-debug location if running with --debug.

Neither of these approaches will work in a Windows release build.

The script is somewhat hard to read, but this is my understanding:

1) It does an optimized build of all components.
2) It does a debug build, again of all components.
3) It packs the optimized binaries.
4) It adds the necessary debug binaries,
   taking them from the debug build tree and renaming file or directory
   to avoid a naming conflict.
These "necessary" binaries are the server and several libraries it needs.

Later, that package will be unpacked and installed somewhere, and tests will be run. At this point, no debug build of "mysqltest" is available.

So if
1) the logic of MTR will remain to pass the "debug=..." to "mysqltest"
   (the client) based on the server being a debug build, *and*
2) an optimized "mysqltest" will continue to fail
   if it is given the "debug=..." option,
then the only solution might be to add the debug "mysqltest" to the package.

It is not needed for any other purpose, and this would be a difference between Windows and Unix packages, so I am not convinced this is a good idea.

Ok so how about mysqltest just ignoring a non-supported debug argument (after outputing a warning message)?  Sounds like that would be the safest solution, assuming mtr doesn't expect any additional output from mysqltest when using --debug.

Yes, I think this is the best solution, "best" as in "least intrusive".

I know it can be debated whether unsupported options should be ignored (like in your proposal, with a warning) or cause an abort, and there are merits to the "abort" principle if the option is essential for the program's function (like supporting  transactions),
but for debugging options I think just giving a warning "unsupported" is the best approach.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/98207

2885 Bjorn Munch	2010-01-26
      Bug #47141 "mysqltest" breaks because it tries to act on a debug option which is disabled
      The debug option is actually ignored, so we just do this unconditionally

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/98218

2885 Bjorn Munch	2010-01-26
      Bug #47141 "mysqltest" breaks because it tries to act on a debug option which is disabled
      Please ignore previous patch, it was mistaken
      Replace --debug with --loose-debug to prevent failure exit

Last fix was too simple, trying to run binlog.binlog_incident I get

G:\bjorn\dbgo\mysql-5.1.43-win-x86\client\relwithdebinfo\\mysqlbinlog.exe: ambiguous option '--loose-loose-debug=d:t:A,......

Some code apparently already adds loose-. Back to the drawing board....

My last fix is not workable due to a my_getopt bug: Bug #50657. I don't see any obvious and simple workaround for that (short of disabling one of the two --debug-xxx options in non-debug mysqlbinlog, which I will not do)

If there is a mysqld-debug installed, mtr.pl will try to find it when the --debug argument is passed to mtr.pl. See code below.

Seems like this rule is only for the mysqld and not the client binaries(which is good I think).

Adding --loose-debug is ok, but I think that even better is to run one of the client binaries and check if it's debug compiled or not. Then only turn on --loose if it's not debug compiled or even don't pass the --debug=.trace argument to them. This would catch any problems with mysqld being debug while the binaries are not.

It's however not feasible to test every single client binary for debug support(I think) -> so maybe add --loose-debug on all clients after all?

The code that prefers mysqld-debug:

sub find_mysqld {
  my ($mysqld_basedir)= @_;

  my @mysqld_names= ("mysqld", "mysqld-max-nt", "mysqld-max",
		     "mysqld-nt");

  if ( $opt_debug ){
    # Put mysqld-debug first in the list of binaries to look for
    mtr_verbose("Adding mysqld-debug first in list of binaries to look for");
    unshift(@mysqld_names, "mysqld-debug");
  }

Yes I know about mysqld-debug. The problem is test with --debug when you have mysqld-debug but not mysqltest-debug.

I don't know if there's any easy and consistent way to test if a client is compiled with debug; seems like the --help output doesn't give any clue. So I think the best option is to try to use debug but in a way which will be silently ignored if the client does not support it.

It would probably be possible to figure out a way to test it, but maybe not worth it. Go ahead with --loose-debug

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/130394

2978 Bjorn Munch	2011-02-04
      Bug #47141 "mysqltest" breaks because it tries to act on a debug option which is disabled
      Replace --debug with --loose-debug to prevent failure exit
      Update: added workaround for 50627, skip all debugging of mysqlbinlog

Pushed to -mtr branches

Pushed into mysql-trunk 5.6.2 (revid:bjorn.munch@oracle.com-20110207235423-p72c50ae0vgdjbdl) (version source revid:bjorn.munch@oracle.com-20110207235315-oejyajxuc8utuwfu) (merge vers: 5.6.2) (pib:24)

Pushed into mysql-5.5 5.5.10 (revid:bjorn.munch@oracle.com-20110207232832-ppycbd3zez7pfvq1) (version source revid:bjorn.munch@oracle.com-20110207232730-yc8qvy7ziu3nd5ds) (merge vers: 5.5.10) (pib:24)

Pushed into mysql-5.1 5.1.56 (revid:bjorn.munch@oracle.com-20110207230245-ignb6bk9zfq1zswp) (version source revid:bjorn.munch@oracle.com-20110207230245-ignb6bk9zfq1zswp) (merge vers: 5.1.56) (pib:24)

Changes to test suite. No changelog entry needed.