MySQL Bugs: #59148: 'INSTALL PLUGIN rpl_semi_sync_master' fails in release build with debug binaries

Bug #59148	'INSTALL PLUGIN rpl_semi_sync_master' fails in release build with debug binaries
Submitted:	24 Dec 2010 7:40	Modified:	9 Feb 2011 19:21
Reporter:	Alexander Nozdrin	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tools: MTR / mysql-test-run	Severity:	S7 (Test Cases)
Version:	5.6.1, 5.5,mysql-trunk	OS:	Any
Assigned to:	Bjørn Munch	CPU Architecture:	Any
Tags:	pb2, test failure

Description:
A few tests fail with the following symptom in release build
with debug binaries:

mysqltest: At line 29: query 'INSTALL PLUGIN rpl_semi_sync_master SONAME '$SEMISYNC_MASTER_PLUGIN'' failed: 1126: Can't open shared library '/export/home2/pb2/test/sb_2-2710269-1293149042.86/usr/lib/mysql/plugin/semisync_master.so' (errno: 2 undefined symbol: my_pthread_fastmutex_lock)

The following tests are affected:
sys_vars.all_vars
sys_vars.rpl_semi_sync_master_enabled_basic
sys_vars.rpl_semi_sync_master_timeout_basic
sys_vars.rpl_semi_sync_master_trace_level_basic
sys_vars.rpl_semi_sync_master_wait_no_slave_basic

The tests fail on the following platforms:
linux2.6-x86-32bit community rpm
linux2.6-x86-32bit community tar-gz
linux2.6-x86-64bit community tar-gz
rhel5-ia64-64bit community rpm
rhel5-x86-32bit community rpm
sles10-ia64-64bit community rpm
sles10-x86-32bit community rpm
sles11-x86-32bit community rpm

Might be related to Bug#49022.

How to repeat:
http://pb2.norway.sun.com/?template=show_pushes&branch=mysql-5.6.1-m5-release

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/128258

3226 Georgi Kodinov	2011-01-10
      Bug #59148 - made tests experimental

rpl.rpl_semi_sync_event and rpl.rpl_semi_sync are also failing with this bug in daily-trunk. See http://pb2.norway.sun.com/?template=mysql_show_test_failure&search=yes&push_id=1950378&tes...

AIUI, the problem is just another effect of the decision to have some internal objects be defined different in debug vs optimized builds.
This base issue is discussed in greater length in bug#49022.

The fix for that bug is to tell a "debug" server on startup (in "mysqld_safe") to search for the plugins in a different directory, by appending "/debug" to the directory given as "--plugin-dir=".
For more details, see the changeset:
http://lists.mysql.com/commits/101974

(Other parts of that bug fix affect packaging, to ensure that the debug plugins are really distributed - this is not relevant here.)

This bug fix seems to work for customers and users.

However, it does not help when we run the test suite: MTR does start the server (be it optimized or debug) withut using "mysqld_safe", so that fix is bypassed.

The clean solution should be to revert that old design of having different mutex object implementations in debug vs optimized builds, in order to get the identical ABI between server and plugin for both build modes (which I strongly propose!).
Until we get that, the workaround seems to be that MTR should provide a modified "--plugin-dir=" option to the debug server.

Summary of this task then is to change MTR so that when passed --debug it looks for debug plugins in a similar manner to mysqld_safe (but needs to support Windows too), and then passes --plugin-dir= to the appropriate mysqld.

Back to the drawing board! This is NOT related to mtr --debug and the debug server. I look at the PB2 logs and see that

1) mtr is not called with --debug
2) The core file is produced by mysqld, not by mysqld-debug

In fact, trying to run mtr --debug on the release build fails due to Bug #47141.

I try to tun manually but can't make those tests fail. This remains a mystery.

OK it has been explained. The mysqld-debug binary was renamed to mysqld before running mtr, possibly as a workaround. mtr still needs to be fixed so that it would work when thing are done the right way using --debug.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/129885

3160 Bjorn Munch	2011-01-28
      Bug #59148 'INSTALL PLUGIN rpl_semi_sync_master' fails in release build with debug binaries
      Do as mysqld_safe: if running mysqld-debug, plugins are in debug subdirs
      NB mtr --debug won't work in this context until 47141 is fixed
      Also moved read_plugin_defs; no point running this in all worker threads

Pushed to 5.5-mtr and trunk-mtr

Pushed into mysql-trunk 5.6.2 (revid:bjorn.munch@oracle.com-20110130115129-js8oloj0kke23y2u) (version source revid:bjorn.munch@oracle.com-20110130115042-u671iduxcr5uw04c) (merge vers: 5.6.2) (pib:24)

Pushed into mysql-5.5 5.5.10 (revid:bjorn.munch@oracle.com-20110130110933-9rcq2ic58w06ae30) (version source revid:bjorn.munch@oracle.com-20110130110933-9rcq2ic58w06ae30) (merge vers: 5.5.10) (pib:24)

Changes to test suite. No changelog entry needed.

Pushed into mysql-5.5 5.5.10 (revid:jonathan.perkin@oracle.com-20110208140736-1173xnoipufbhowh) (version source revid:jonathan.perkin@oracle.com-20110208135903-jhzy6wq16b2fx7pg) (merge vers: 5.5.10) (pib:24)

Pushed into mysql-trunk 5.6.2 (revid:georgi.kodinov@oracle.com-20110208155412-tfy4l5hqxi0g7o41) (version source revid:georgi.kodinov@oracle.com-20110208154951-gzqgr74u4bndswi8) (merge vers: 5.6.2) (pib:24)