Bug #69265 -DBUILD_CONFIG=mysql_release -DWITH_DEBUG=ON fails 4 and skips 27 MTR tests
Submitted: 17 May 2013 12:43 Modified: 19 Aug 2013 17:24
Reporter: Laurynas Biveinis (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:Tools: MTR / mysql-test-run Severity:S7 (Test Cases)
Version:5.6.11 OS:Any
Assigned to: CPU Architecture:Any
Tags: mtr cmake

[17 May 2013 12:43] Laurynas Biveinis
Description:
Setting -DBUILD_CONFIG=mysql_release -DWITH_DEBUG=ON breaks most if
not all the plugin tests in the testsuite, most of them by skipping,
but some by failing.  The failing tests might be buggy in themselves,
i.e. they attempt to run when the required plugins are absent, but in
this case this was what helped to detect this bug.

Other issues here. For now I am not reporting them as separate bugs.
- Skip reason: "Test requires: 'true'". Why, I certainly have /bin/true on my system!
- The four failing tests should not be failing but skipped if no required plugin built.

The skipped tests:

rpl.rpl_semi_sync_event 'mix'            w5 [ skipped ]  Need semisync plugins
rpl.rpl_semi_sync_event 'stmt'           w4 [ skipped ]  Need semisync plugins
rpl.rpl_mixed_implicit_commit_binlog 'mix' w7 [ skipped ]  UDF requires the environment variable \$UDF_EXAMPLE_LIB to be set (normally done by mtr)
rpl.rpl_udf 'mix'                        w2 [ skipped ]  UDF requires the environment variable \$UDF_EXAMPLE_LIB to be set (normally done by mtr)
rpl.rpl_udf 'row'                        w3 [ skipped ]  UDF requires the environment variable \$UDF_EXAMPLE_LIB to be set (normally done by mtr)
rpl.rpl_stm_implicit_commit_binlog 'stmt' w1 [ skipped ]  UDF requires the environment variable \$UDF_EXAMPLE_LIB to be set (normally done by mtr)
rpl.rpl_udf 'stmt'                       w8 [ skipped ]  UDF requires the environment variable \$UDF_EXAMPLE_LIB to be set (normally done by mtr)
rpl.rpl_row_implicit_commit_binlog 'row' w6 [ skipped ]  UDF requires the environment variable \$UDF_EXAMPLE_LIB to be set (normally done by mtr)
binlog.binlog_unsafe 'stmt'              w3 [ skipped ]  UDF requires the environment variable \$UDF_EXAMPLE_LIB to be set (normally done by mtr)
sys_vars.rpl_semi_sync_master_enabled_basic w4 [ skipped ]  Need semisync plugins
main.fulltext_plugin                     w7 [ skipped ]  simple parser requires the environment variable \$SIMPLE_PARSER to be set (normally done by mtr)
sys_vars.rpl_semi_sync_master_timeout_basic w4 [ skipped ]  Need semisync plugins
sys_vars.rpl_semi_sync_master_trace_level_basic w4 [ skipped ]  Need semisync plugins
sys_vars.rpl_semi_sync_master_wait_no_slave_basic w4 [ skipped ]  Need semisync plugins
sys_vars.rpl_semi_sync_slave_enabled_basic w4 [ skipped ]  Need semisync plugins
sys_vars.rpl_semi_sync_slave_trace_level_basic w4 [ skipped ]  Need semisync plugins
main.udf                                 w3 [ skipped ]  UDF requires the environment variable \$UDF_EXAMPLE_LIB to be set (normally done by mtr)
main.udf_skip_grants                     w7 [ skipped ]  UDF requires the environment variable \$UDF_EXAMPLE_LIB to be set (normally done by mtr)
main.validate_password_plugin            w6 [ skipped ]  validate password requires the environment variable \$VALIDATE_PASSWORD to be set (normally done by mtr)
main.wl6443_deprecation                  w6 [ skipped ]  validate password requires the environment variable \$VALIDATE_PASSWORD to be set (normally done by mtr)
main.plugin_auth_qa_2                    w7 [ skipped ]  Test requires: 'true'
perfschema.hostcache_ipv6_auth_plugin    w3 [ skipped ]  Test requires: 'true'
main.plugin_auth                         w7 [ skipped ]  Test requires: 'true'
main.mysql_client_test                   w3 [ skipped ]  Test requires: 'true'
main.plugin_auth_qa                      w7 [ skipped ]  Test requires: 'true'
main.plugin_auth_qa_1                    w7 [ skipped ]  Test requires: 'true'
perfschema.hostcache_ipv4_auth_plugin    w7 [ skipped ]  Test requires: 'true'
main.plugin_auth_qa_3                    w7 [ skipped ]  Test requires: 'true'

The failing tests:

main.audit_plugin                        w7 [ fail ]
        Test ended at 2013-05-09 06:50:27

CURRENT_TEST: main.audit_plugin
mysqltest: At line 38: query 'INSTALL PLUGIN audit_null SONAME 'adt_null.$expected_extension'' failed with wrong errno 1126: 'Can't open shared library '/mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-raring-64bit/Percona-Server/lib/plugin/adt_null.so' (errno: 2 /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-raring-64bit/Percona-Server/lib/plugin/adt_null.so: cannot )', instead of 1127...

main.multi_plugin_load_add2              w7 [ fail ]
        Test ended at 2013-05-09 06:51:00

CURRENT_TEST: main.multi_plugin_load_add2
--- /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-raring-64bit/Percona-Server/mysql-test/r/multi_plugin_load_add2.result	2013-05-08 10:35:57.000000000 +0300
+++ /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-raring-64bit/Percona-Server/mysql-test/r/multi_plugin_load_add2.reject	2013-05-09 09:50:59.000000000 +0300
@@ -8,7 +8,5 @@
 WHERE PLUGIN_NAME IN ('test_plugin_server', 'qa_auth_server')
 ORDER BY 1;
 PLUGIN_NAME	PLUGIN_STATUS
-qa_auth_server	ACTIVE
-test_plugin_server	ACTIVE
 SELECT * FROM mysql.plugin ORDER BY name;
 name	dl

mysqltest: Result length mismatch

main.multi_plugin_load_add               w3 [ fail ]
        Test ended at 2013-05-09 06:51:19

CURRENT_TEST: main.multi_plugin_load_add
--- /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-raring-64bit/Percona-Server/mysql-test/r/multi_plugin_load_add.result	2013-05-08 10:35:57.000000000 +0300
+++ /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/debug/Host/ubuntu-raring-64bit/Percona-Server/mysql-test/r/multi_plugin_load_add.reject	2013-05-09 09:51:17.000000000 +0300
@@ -8,7 +8,5 @@
 WHERE PLUGIN_NAME IN ('test_plugin_server', 'qa_auth_server')
 ORDER BY 1;
 PLUGIN_NAME	PLUGIN_STATUS
-qa_auth_server	ACTIVE
-test_plugin_server	ACTIVE
 SELECT * FROM mysql.plugin ORDER BY name;
 name	dl

mysqltest: Result length mismatch

main.bug12969156                         w5 [ fail ]
        Test ended at 2013-05-09 06:58:23

CURRENT_TEST: main.bug12969156
mysqltest: At line 13: Can't initialize replace from 'replace_result $DAEMONEXAMPLE DAEMONEXAMPLE'

The result from queries just before the failure was:
#
# Bug #12969156 : SEGMENTATION FAULT ON UNINSTALLING
#  DAEMON_EXAMPLE PLUGIN
#

How to repeat:
cmake .. -DBUILD_CONFIG=mysql_release -DWITH_DEBUG=ON -DWITH_EMBEDDED_SERVER=OFF

cd mysql-test

./mysql-test-run --debug-server any of above tests

Suggested fix:
This is all caused by MTR not finding any plugin in mysql-test/include/plugin.defs and that is caused in find_plugin() in mysql-test-run.pl:

    # If running debug server, plugins will be in 'debug' subdirectory
    $plug_file= "debug/$plug_file" if $running_debug;

which is not true, there is no debug subdirectory.

The fix is trivial, assuming that the bug is not the lack of debug directory in the first place:

=== modified file 'mysql-test/mysql-test-run.pl'
--- mysql-test/mysql-test-run.pl	2013-02-20 09:31:47 +0000
+++ mysql-test/mysql-test-run.pl	2013-05-17 12:26:15 +0000
@@ -2197,16 +2197,10 @@
 sub read_plugin_defs($)
 {
   my ($defs_file)= @_;
-  my $running_debug= 0;
 
   open(PLUGDEF, '<', $defs_file)
     or mtr_error("Can't read plugin defintions file $defs_file");
 
-  # Need to check if we will be running mysqld-debug
-  if ($opt_debug_server) {
-    $running_debug= 1 if find_mysqld($basedir) =~ /mysqld-debug/;
-  }
-
   while (<PLUGDEF>) {
     next if /^#/;
     my ($plug_file, $plug_loc, $plug_var, $plug_names)= split;
@@ -2214,9 +2208,6 @@
     next unless $plug_file;
     mtr_error("Lines in $defs_file must have 3 or 4 items") unless $plug_var;
 
-    # If running debug server, plugins will be in 'debug' subdirectory
-    $plug_file= "debug/$plug_file" if $running_debug;
-
     my ($plugin)= find_plugin($plug_file, $plug_loc);
 
     # Set env. variables that tests may use, set to empty if plugin

All of the tests above start passing then.
[20 May 2013 8:14] Laurynas Biveinis
I forgot to consider the option of running the testsuite on an installed server.

The MTR code appears to work in that case and not for running the testsuite in the source tree.  The proposed fix would reverse the situation and thus it's a wrong fix.
[23 May 2013 6:25] Laurynas Biveinis
A better fix is (ignore the "Percona-Server" in the path).

Installed server testsuite runs keep their behavior (didn't actually test), in-source testsuite runs are fixed.

=== modified file 'Percona-Server/mysql-test/mysql-test-run.pl'
--- Percona-Server/mysql-test/mysql-test-run.pl	2013-05-12 09:13:00 +0000
+++ Percona-Server/mysql-test/mysql-test-run.pl	2013-05-23 06:22:07 +0000
@@ -2226,7 +2226,7 @@
     mtr_error("Lines in $defs_file must have 3 or 4 items") unless $plug_var;
 
     # If running debug server, plugins will be in 'debug' subdirectory
-    $plug_file= "debug/$plug_file" if $running_debug;
+    $plug_file= "debug/$plug_file" if $running_debug && !$source_dist;
 
     my ($plugin)= find_plugin($plug_file, $plug_loc);
[6 Jun 2013 5:36] Prabeen Kumar Pradhan
Thank you for the bug report. Verified as described
[18 Jun 2013 14:40] Paul DuBois
Changes for test suite. No changelog entry needed.
[1 Aug 2013 15:56] Laurynas Biveinis
It seems that the proposed fix was incorrect:

laurynas@laurynas-ThinkPad-T410:~/percona/src/5.5$ bzr log -r 4378
------------------------------------------------------------
revno: 4378
committer: sayantan dutta <sayantan.dutta@oracle.com>
branch nick: mysql-5.5
timestamp: Fri 2013-06-07 19:29:56 +0530
message:
  Bug #16917425  -DBUILD_CONFIG=MYSQL_RELEASE -DWITH_DEBUG=ON FAILS 4 AND SKIPS 27 MTR TESTS
laurynas@laurynas-ThinkPad-T410:~/percona/src/5.5$ bzr log -r 4379
------------------------------------------------------------
revno: 4379
committer: Olav Sandstaa <olav.sandstaa@oracle.com>
branch nick: 5.5-fix
timestamp: Fri 2013-06-07 21:34:34 +0200
message:
  Bug #16917425  -DBUILD_CONFIG=MYSQL_RELEASE -DWITH_DEBUG=ON FAILS 4 AND SKIPS 27 MTR TESTS
  
  Revert fix since it caused mtr tests to not run in pushbuild.
[2 Aug 2013 10:18] Prabeen Kumar Pradhan
This issue was fixed on 18-Jun-13. Kindly check and confirm if you still see this issue.
[2 Aug 2013 10:22] Laurynas Biveinis
Prabeen -

Could you please point me at the bzr revision? (or even the tree if it was not fixed on 5.5 perhaps). I see only the following revisions for Jun 18th. It's obviously not 4393, but maybe one of two mysterious other revisions?

------------------------------------------------------------
revno: 4393
committer: Vasil Dimov <vasil.dimov@oracle.com>
branch nick: mysql-5.5
timestamp: Tue 2013-06-18 17:12:28 +0300
message:
  Fix Bug#16907783 5.5 STILL CRASHES IN DICT_UPDATE_STATISTICS WITH CONCURRENT
  DDL AND I_S QUERIES
  
  Skip partially created indexes (ones whose name starts with TEMP_INDEX_PREFIX)
  from stats gathering.
  
  Because InnoDB reports HA_INPLACE_ADD_INDEX_NO_WRITE to MySQL, the latter
  allows parallel execution of ha_innobase::add_index() and ha_innobase::info().
  
  Reviewed by:  Inaam (rb:2613)
------------------------------------------------------------
revno: 4392 [merge]
committer: mysql-builder@oracle.com
branch nick: clone
timestamp: Tue 2013-06-18 15:49:13 +0530
message:
  
------------------------------------------------------------
revno: 4391
committer: mysql-builder@oracle.com
branch nick: clone
timestamp: Tue 2013-06-18 10:20:30 +0530
message:
  
------------------------------------------------------------
[2 Aug 2013 11:24] Prabeen Kumar Pradhan
Below is the revision detail in which it was fixed:
http://bazaar.launchpad.net/~mysql/mysql-server/5.6/revision/5217
[5 Aug 2013 3:39] Laurynas Biveinis
The bug is _not_ fixed: the referenced fix merely replaces one failure mode with another, causing the tests to be skipped instead of failed, because the root cause of plugins being searched for in the wrong dirs is not addressed.

Thus now instead of 4 failing and 27 skipped tests we have 31 skipped test instead:

$ ./mysql-test-run --mem --debug audit_plugin
...
main.audit_plugin                        [ skipped ]  Audit_null plugin requires the environment variable \$AUDIT_NULL to be set (normally done by mtr)
...

etc.
[6 Aug 2013 12:11] Paul DuBois
Changes for test suite. No changelog entry needed.
[6 Aug 2013 13:05] Prabeen Kumar Pradhan
We are currently working on this. Will update progress.
[8 Aug 2013 5:25] Anitha Gopi
Laurynas,
Please upload your suggested patch under contributions tag.
[8 Aug 2013 12:36] Laurynas Biveinis
The bug tracker says

"Contributions can be accepted to Open bugs only."
[8 Aug 2013 12:52] Laurynas Biveinis
bug 69963.
[12 Aug 2013 9:27] Laurynas Biveinis
Bug 69265 patch

(*) I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

Contribution: bug69265.patch (text/x-patch), 572 bytes.

[12 Aug 2013 9:27] Anitha Gopi
I have set status to Open. Please try now.
[12 Aug 2013 9:29] Anitha Gopi
Got it now. Thanks.
[19 Aug 2013 17:24] Paul DuBois
Changes for test suite. No changelog entry needed.

Thanks for Laurynas Biveinis for the patch to fix this issue.