MySQL Bugs: #41502: MTR v2 should not load plugins during server bootstrap

Bug #41502	MTR v2 should not load plugins during server bootstrap
Submitted:	16 Dec 2008 13:06	Modified:	30 Jan 2009 18:04
Reporter:	John Embretsen	Email Updates:
Status:	Closed	Impact on me:	None
Category:	MySQL Server: Tests	Severity:	S7 (Test Cases)
Version:	MTR_v2, 6.0-rpl, 5.1-rpl	OS:	Any
Assigned to:	John Embretsen	CPU Architecture:	Any
Tags:	51rpl, 6.0-rpl-green, falcon_bug_39708, pushbuild

Description:
The new version of the mysql-test-run.pl script (to be referred to as "MTR v2") (currently only in replication branches) runs mysqld -bootstrap differently than the current MTR (in 6.0-main and non-rpl sub-branches).

One of the differences is that the new MTR no longer skips the Falcon storage engine during bootstrap, which results in

  a) Static variables (e.g. page size) being initialized with default values.
  b) Tests relying on setting non-default values of static plugin (in this case Falcon) options will fail (see Bug#41014).

There may be other side-effects as well, yet to be discovered.

Main implementor of MTR v2, Magnus Svensson, recently said:

"the features we use during bootstrap should only rely on what is builtin to mysqld"

Based on the mysql-6.0 and mysql-6.0_with_tests_from_6.0-rpl branches, here is a sample diff between MTR_v1 and MTR_v2:

----------> MTR mysqld bootstrap option diff start

+  mtr_report("Installing system database...");

   my $args;
-  my $cmd_args;
   mtr_init_args(\$args);
   mtr_add_arg($args, "--no-defaults");
   mtr_add_arg($args, "--bootstrap");
-  mtr_add_arg($args, "--basedir=%s", $path_my_basedir);
-  mtr_add_arg($args, "--datadir=%s", $data_dir);
+  mtr_add_arg($args, "--basedir=%s", $install_basedir);
+  mtr_add_arg($args, "--datadir=%s", $install_datadir);
   mtr_add_arg($args, "--loose-skip-innodb");
   mtr_add_arg($args, "--loose-skip-ndbcluster");
-  mtr_add_arg($args, "--loose-skip-falcon");
-  mtr_add_arg($args, "--disable-sync-frm");
-  mtr_add_arg($args, "--loose-disable-debug");
-  mtr_add_arg($args, "--tmpdir=.");
+  mtr_add_arg($args, "--tmpdir=%s", "$opt_vardir/tmp/");
   mtr_add_arg($args, "--core-file");

----------> mysqld bootstrap option diff end

We see that the option --loose-skip-falcon (among others) has been removed.
This is around line 2591 in the mysql-test-run.pl script in the mysql-6.0-rpl branch, in the "mysql_install_db" function.

This was discovered during analysis of Bug#41014 (falcon_bug_39708 fails in pushbuild in 6.0-rpl: "succeeded - should have failed"). Fixing Bug#41014 (thus also merging mysql-6.0-rpl into mysql-6.0) depends on this bug.

How to repeat:
Branch the mysql-6.0-rpl branch (or equivalent) to acquire MTR v2.

cd mysql-test

Run: ./mysql-test-run.pl --suite=falcon falcon_bug_39708 --verbose --verbose

Examine output near "Installing system database...". Verify that the option --loose-skip-falcon is not included in the mysqld command line which includes the --bootstrap option.

The test will fail, mainly because it depends on an options file setting the option --falcon-page-size, which is static and has already been initialized (to default) during bootstrap.

To compare with MTR v1, branch the mysql-6.0 branch and run the same test:

cd mysql-test

Run: ./mysql-test-run.pl --suite=falcon --script-debug falcon_bug_39708

The test will pass. The main difference is that the --loose-skip-falcon option was included in the bootstrap command, making it possible to initialize Falcon while running the test suite, using non-default static options if necessary.

Note that bootstrap options are the same regardless of test case and test suite.

Suggested fix:
Option 1:

 - Add --loose-skip-falcon to bootstrap options in MTR v2 (function mysql_install_db).
 - It might be a good idea to include --loose-skip-maria as well, to avoid similar issues with the Maria storage engine going forward.

Option 2:

 - Remove current --loose-skip-* bootstrap options and add an option which disables all plugins instead.

Please fix by suggested solution 2

I tried using suggested Option 2: --skip-plugin-load 

However this does not seem to resolve the issue (tested by running falcon_bug_39708 test). 

mysql-test/var/log/bootstrap.log shows an error message (replaced full paths with "$srcdir" for readability):

$srcdir/sql/mysqld --no-defaults --bootstrap --basedir=$srcdir --datadir=$srcdir/mysql-test/var/install.db --skip-plugin-load --tmpdir=$srcdir/mysql-test/var/tmp/ --core-file --language=$srcdir/sql/share/english --character-sets-dir=$srcdir/sql/share/charsets

081216 14:51:06 [ERROR] Can't open shared library '$srcdir/lib/mysql/plugin/0' (errno: 2 cannot open shared object file: No such file or directory)
081216 14:51:06 [Note] Falcon: unable to open system data files.
081216 14:51:06 [Note] Falcon: creating new system data files.
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
(...)

The lines mentioning Falcon indicate that the Falcon plugin is indeed being initialized.

Advice would be appreciated (I know nothing about this option; it does not seem to be documented). The binaries were configured with the following options:

CFLAGS="-g" CXXFLAGS="-g"
--with-debug=full
--with-ssl
--with-plugins=max
--with-extra-charsets=complex
--with-embedded-server

Note that adding --loose-skip-falcon instead (fix Option 1) seems to work, though.

Patch against mysql-6.0-rpl implementing Suggested Fix Option 1:

=== modified file 'mysql-test/mysql-test-run.pl'
--- mysql-test/mysql-test-run.pl        2008-12-15 12:41:31 +0000
+++ mysql-test/mysql-test-run.pl        2008-12-16 12:49:04 +0000
@@ -2589,6 +2589,8 @@
   mtr_add_arg($args, "--basedir=%s", $install_basedir);
   mtr_add_arg($args, "--datadir=%s", $install_datadir);
   mtr_add_arg($args, "--loose-skip-innodb");
+  mtr_add_arg($args, "--loose-skip-falcon");
   mtr_add_arg($args, "--loose-skip-ndbcluster");
   mtr_add_arg($args, "--tmpdir=%s", "$opt_vardir/tmp/");
   mtr_add_arg($args, "--core-file");

Assigning myself since I am already working on this.
Setting Magnus as reviewer since he is the MTR expert at the moment.
Setting Sven as second reviewer since a fix for this issue needs to be pushed to rpl branches (where MTR v2 is) and he is the one who reported the original Bug#41014.

Please commit a patch with this solution. I checked the code in mysqld.cc where we call 'plugin_init' and there are no trace of a possibility to disable the loading pf plugins(except by using --help). In my opinion there should be such a flag or --bootstrap should also pass the flag PLUGIN_INIT_SKIP_INITIALIZATION to 'plugin_init' like --help does. But that is another bug.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61841

2708 John H. Embretsen	2008-12-17
      Fix (preliminary?) for Bug#41502 - MTR v2 should not load plugins during server bootstrap.
      
      Adding --loose-skip-falcon option to the mysqld options provided by MTR (v2) during mysqld bootstrap in order to avoid plugin (in this case Falcon) initialization of static variables etc. Options --loose-skip-innodb and --loose-skip-ndbcluster were already included.
      
      This will fix Bug#41014 (falcon_bug_39708 fails in pushbuild in 6.0-rpl: "succeeded - should have failed")
      in the case of MTR v2 (which currently is available in -rpl branches only). 
      MTR v1 (e.g. in main 6.0 branch) does not have this problem.
      
      It would be more ideal to remove the --loose-skip-* options and provide a single option disabling all plugin initialization instead, or have bootstrap do this by default. Server modifications are (most likely) needed to be able to do that.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61963

2755 John H. Embretsen	2008-12-18 [merge]
      Fix (preliminary?) for Bug#41502 - MTR v2 should not load plugins during server
      bootstrap.
            
      Adding --loose-skip-falcon option to the mysqld options provided by MTR (v2) during
      mysqld bootstrap in order to avoid plugin (in this case Falcon) initialization of static
      variables etc. Options --loose-skip-innodb and --loose-skip-ndbcluster were already
      included.
            
      This will fix Bug#41014 (falcon_bug_39708 fails in pushbuild in 6.0-rpl: "succeeded
      - should have failed")
      in the case of MTR v2 (which currently is available in -rpl branches only). 
      MTR v1 (e.g. in main 6.0 branch) does not have this problem.
            
      It would be more ideal to remove the --loose-skip-* options and provide a single
      option disabling all plugin initialization instead, or have bootstrap do this by default.
      Server modifications are (most likely) needed to be able to do that.

Fix ("Option 1", --loose-skip-falcon) pushed to:
 mysql-6.0-rpl (revision-id: john.embretsen@sun.com-20081218142940-v0w20qc3a6pxqey6)
 mysql-5.1-rpl (revision-id: john.embretsen@sun.com-20081218145516-sosg2436ne36yflq)

Magnus, I'll leave it up to you to file a new bug for adding a flag for skipping plugin initialization and/or change --bootstrap as you mention in your comment on Dec 16 if you want to, because you obviously know more about this than I do :)

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/62605

2709 He Zhenxing	2009-01-07 [merge]
      Auto merge

Pushed into 6.0.10-alpha (revid:luis.soares@sun.com-20090129165607-wiskabxm948yx463) (version source revid:luis.soares@sun.com-20090129163120-e2ntks4wgpqde6zt) (merge vers: 6.0.10-alpha) (pib:6)

Pushed into 5.1.32 (revid:luis.soares@sun.com-20090129165946-d6jnnfqfokuzr09y) (version source revid:zhenxing.he@sun.com-20090107142538-aeoxq3lddk0g4uff) (merge vers: 5.1.31) (pib:6)

Test suite changes. No changelog entry needed.

Pushed into 5.1.32-ndb-6.3.23 (revid:tomas.ulin@sun.com-20090217131017-6u8qz1edkjfiobef) (version source revid:tomas.ulin@sun.com-20090203133556-9rclp06ol19bmzs4) (merge vers: 5.1.32-ndb-6.3.22) (pib:6)

Pushed into 5.1.32-ndb-6.4.3 (revid:tomas.ulin@sun.com-20090217134419-5ha6xg4dpedrbmau) (version source revid:tomas.ulin@sun.com-20090203133556-9rclp06ol19bmzs4) (merge vers: 5.1.32-ndb-6.3.22) (pib:6)

Pushed into 5.1.32-ndb-6.2.17 (revid:tomas.ulin@sun.com-20090217134216-5699eq74ws4oxa0j) (version source revid:tomas.ulin@sun.com-20090201210519-vehobc4sy3g9s38e) (merge vers: 5.1.32-ndb-6.2.17) (pib:6)