| Bug #49022 | Plugins included into bin release cannot be installed on debug version of server | ||
|---|---|---|---|
| Submitted: | 24 Nov 2009 5:15 | Modified: | 23 Aug 2010 13:16 |
| Reporter: | Elena Stepanova | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server | Severity: | S3 (Non-critical) |
| Version: | 5.1.41, 5.5.0 | OS: | Any |
| Assigned to: | Mats Kindahl | CPU Architecture: | Any |
[24 Nov 2009 5:15]
Elena Stepanova
[24 Nov 2009 8:11]
MySQL Verification Team
at least for innodb plugin, you have to use the debug version of the plugin with the debug version of the server. it's in .\lib\plugin\debug
[24 Nov 2009 9:09]
Sveta Smirnova
Thank you for the report. Verified as described: Linux *tar.gz packages have no ./libs/plugin/debug
[24 Nov 2009 9:31]
Daniel Fischer
I guess this is what happens when you add a feature to a GA release. We have two options: 1) Include debug builds of plugins in our packages, in the same directory where non-debug builds go, with "-debug" or a similar tag in the file name. This means that test cases that load plugins need to treat debug and non-debug servers differently and code that uses INSTALL PLUGIN won't necessarily be compatible between debug and non-debug builds of mysqld. 2) Include debug builds of plugins in a separate directory, but with the same file names as non-debug builds. This means we need to tell the debug server to look for plugins in a different place by default, and all tools that can potentially start mysqld in debug mode need to be aware of this; mtr will have to learn not to pass the non-debug plugin_dir value as well. 3) There is no third option, as I said we only have two.
[24 Nov 2009 19:52]
Sergei Golubchik
third option - always use vanilla mutexes in plugins
[24 Nov 2009 19:53]
Sergei Golubchik
fourth option is WL#5134 - but it's not even in Betony
[17 Dec 2009 11:31]
Joerg Bruehe
As agreed with Jonathan, I grabbed this bug. I have discussed it with Serg in IRC, and the solution seems to be that plugins should use a mutex implementation which is always available (= in both servers). One way to achieve that would be to always include the "*fastmutex*()" functions, even in a debug server, so it would be unused code in the debug binary unless a plugin (always optimized) is loaded. The opposite way, to always include the SAFE_MUTEX implementation, would force us to compile the plugins using SAFE_MUTEX, which might be bad for performance. The third way is to always use the "vanilla" implementation, the system library functions, and this is what Serg proposes. For this, here is my patch: --- start quote --- === modified file 'include/my_pthread.h' --- include/my_pthread.h 2008-10-15 22:28:26 +0000 +++ include/my_pthread.h 2009-12-16 23:39:08 +0000 @@ -516,7 +516,11 @@ void safe_mutex_end(FILE *file); #define safe_mutex_assert_not_owner(mp) #endif /* SAFE_MUTEX */ -#if defined(MY_PTHREAD_FASTMUTEX) && !defined(SAFE_MUTEX) +#if defined(MY_PTHREAD_FASTMUTEX) && !defined(SAFE_MUTEX) && !defined(MYSQL_DYNAMIC_PLUGIN) +/* Plugins must not call "*fastmutex*()", because these functions + * will be provided in an optimized server only. + * If the plugin uses the system-provided mutex functions, + * it can also be loaded into a debug server (bug#49022). */ typedef struct st_my_pthread_fastmutex_t { pthread_mutex_t mutex; @@ -547,7 +551,7 @@ int my_pthread_fastmutex_lock(my_pthread #define pthread_cond_timedwait(A,B,C) pthread_cond_timedwait((A),&(B)->mutex,(C)) #define pthread_mutex_trylock(A) pthread_mutex_trylock(&(A)->mutex) #define pthread_mutex_t my_pthread_fastmutex_t -#endif /* defined(MY_PTHREAD_FASTMUTEX) && !defined(SAFE_MUTEX) */ +#endif /* defined(MY_PTHREAD_FASTMUTEX) && !defined(SAFE_MUTEX) && !defined(MYSQL_DYNAMIC_PLUGIN) */ /* READ-WRITE thread locking */ --- end quote --- This builds fine, the InnoDB plugin loads, but it causes a reproducible timeout in both optimized and debug runs: Test "innodb_mysql" (only) hits the timeout, and it seems to include some polling loop: In the 15 minute timeout period, the server accumulates more than 6 minutes of CPU time. This happens when using the builtin InnoDB already, then similar with the plugin. I will attach a stack backtrace (builtin InnoDB).
[17 Dec 2009 11:34]
Joerg Bruehe
Stack backtrace: optimized build, test "innodb_mysql", builtin InnoDB, timeout.
Attachment: backtrace-49022.txt (text/plain), 6.59 KiB.
[21 Dec 2009 15:51]
Joerg Bruehe
I tried my patch in 5.5, using the sources of 5.5.0-m2.
Here, the InnoDB plugin is not involved but rather the semisync plugin is.
A normal build + test passed, but a mix of debug server and optimized plugin failed because they share objects whose implementation differs -
this is the output:
=====
rpl.rpl_semi_sync 'row' [ fail ]
Test ended at 2009-12-21 16:11:50
CURRENT_TEST: rpl.rpl_semi_sync
mysqltest: At line 96: query 'create table t1 (a int) engine=$engine_type' failed: 2013: Lost connection to MySQL server during query
The result from queries just before the failure was:
< snip >
rpl_semi_sync_master_enabled OFF
[ enable semi-sync on master ]
set global rpl_semi_sync_master_enabled = 1;
show variables like 'rpl_semi_sync_master_enabled';
Variable_name Value
rpl_semi_sync_master_enabled ON
[ status of semi-sync on master should be ON even without any semi-sync slaves ]
show status like 'Rpl_semi_sync_master_clients';
Variable_name Value
Rpl_semi_sync_master_clients 0
show status like 'Rpl_semi_sync_master_status';
Variable_name Value
Rpl_semi_sync_master_status ON
show status like 'Rpl_semi_sync_master_yes_tx';
Variable_name Value
Rpl_semi_sync_master_yes_tx 0
#
# BUG#45672 Semisync repl: ActiveTranx:insert_tranx_node: transaction node allocation failed
# BUG#45673 Semisynch reports correct operation even if no slave is connected
#
More results from queries before failure can be found in /export/home/my/tmp-200912201700-5.5.0-m2-bug49022-22154/linux-x86_64/test/mysql-test/var/log/rpl_semi_sync.log
Server [mysqld.1 - pid: 26443, winpid: 26443, exit: 256] failed during test run
Server log from this test:
091221 18:11:49 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
091221 18:11:49 InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
091221 18:11:49 InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
091221 18:11:49 InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
091221 18:11:50 InnoDB Plugin 1.0.5 started; log sequence number 0
091221 18:11:50 [Note] Event Scheduler: Loaded 0 events
091221 18:11:50 [Note] /export/home/my/tmp-200912201700-5.5.0-m2-bug49022-22154/linux-x86_64/test/bin/mysqld: ready for connections.
Version: '5.5.0-m2-bug49022-debug-log' socket: '/tmp/xPsCRZ59SA/mysqld.1.sock' port: 13000 MySQL Community Server - Debug (GPL)
091221 18:11:50 [Note] Start binlog_dump to slave_server(2), pos(, 4)
091221 18:11:50 [Note] Semi-sync replication initialized for 151 transactions.
091221 18:11:50 [Note] Semi-sync replication enabled on the master.
mysqld: sql_class.h:1888: const char* THD::enter_cond(pthread_cond_t*, safe_mutex_t*, const char*): Assertion `(mutex)->count > 0 && pthread_equal(pthread_self(), (mutex)->thread)' failed.
091221 18:11:50 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
=====
My conclusion is:
There are synchronization objects (mutexes, maybe others) which are accessed by both server code and plugin code, so the functions accessing them should be considered part of the ABI for plugins.
Our current approach of
- building a "debug" server and renaming it,
- building everything optimized,
- combining these optimized binaries and the debug server
does not work as long as the functions accessing mutexes (and the data structures used) are different in a debug and an optimized build.
[21 Dec 2009 16:30]
Sergei Golubchik
Joerg, I think you can solve MySQL problem (undefined symbols), and open another bug for semisync plugin. It should not use thd_enter_cond() - in fact, remove this function completely, as it was introduced for semisync - because it doesn't work (that is, does not isolate server implementation from the plugin).
[5 Jan 2010 19:45]
Joerg Bruehe
Unassign myself, this is way beyond packaging. Proper category would be "Plugin ABI", but we don't have that.
[12 Jan 2010 7:38]
Zhenxing He
IMHO, I think avoiding using ABIs that are different in optimized build and debug build for all plugins is not a trivial task, especially for innodb and other storage engine plugins, which used mutex directly and also dependent on mutex indirectly by accessing other objects that included mutex members such as TABLE_SHARE, THR_LOCK, etc. Accessing these objects in the plugin of different build type will result in error or crash. It is also not an easy task to remove the call to thd_enter_cond for semisync. So I think we should either make ABIs compatible in all build mode, or always loading the plugins of the same build type.
[12 Jan 2010 8:56]
Zhenxing He
For semisync plugins, one solution could be always build the plugin using the vanilla mutex, and meanwhile make fastmutex, safemutex support vanilla mutex too. Unfortunately this solution does not work for innodb plugin.
[12 Jan 2010 9:24]
Mats Kindahl
I think it is necessary that ABIs are stable for all kind of build types so that it is possible to load non-debug plugins into debug servers, and the opposite. In this respect, we need to support Serg's fourth option. However, for this particular case, my_pthread_fastmutex_lock is not part of an ABI (simply because there is no ABI for mutexes), so I think that we need to provide a debug-build version of the semi-sync plugin. Whenever a plug-in is accessing the internals of the server, all bets are off and the only way to guarantee that the same functions and data-structures are used is to build the server and the plug-in with the same configuration options and the same version of the server. Especially note that it is *not* sufficient to only use vanilla mutexes in plugins, because this is a problem with the server and the plugin accessing the same mutex, so it is necessary that also the mutexes in the server are vanilla. Since we cannot guarantee the type of any particular mutex that is not part of an ABI, this would mean that *all* mutexes are of the vanilla kind, which basically prohibits the use of fastmutex. So, to solve this bug, I would suggest that a debug version of the semi-sync plugin is provided with the release. Since the mysql version is provided when starting the server, my personal opinion is that Daniel's option 2 would be the best one since a user may not be aware of what server is started and hence should not have to make the decision on whether to load the debug version of the semi-sync or not, it is something that should be set up properly when starting the server using mysqld_safe.
[13 Jan 2010 11:21]
Joerg Bruehe
I do not share Mats' opinion claiming this to be no ABI problem because there were no ABI for mutexes. For me, any call between separate entities (like server and plugin) uses/passes an ABI, and the fault is that the written specification (wherever it may be) does not include such mutex calls. But that is quibbling about words, not really helping to fix the bug. IMO, the question is: a) Can we restrict plugins such that they never call server functions (like mutex functions) on any object which the server also uses? (In that case, it would be sufficient the functions are provided by the server, and it would be the plugin's duty to create and destroy the object. As any such object would then be used only by the server or only by the plugin but never by both, different implementations (compile modes) would not matter.) OR b) Can we change the mutex function interface such that on any platform, their ABI is stable, regardless of the compilation mode? (This would probably mean to pick one and only one of the "vanilla", "SAFE_MUTEX", or "fastmutex" interfaces, to ensure that the data structure they operate on is identical for debug and optimized builds, and to restrict the additional debug code, if any, to only evaluate and report fields, maybe log calls, but accept that a debug server can be called from an optimized plugin.) OR c) Is it really necessary to provide both a debug and an optimized plugin, and make sure the server always loads the matching variety? I have listed the alternatives in the order of my technical preference: - Dedicated objects (a) would give the greatest flexibility, - stable interface (b) would impose restrictions on debug code, - separate debug plugins (c) would require changes to the directory structure (or plugin naming) and to the various start/management/stop scripts. If this were only about packaging, the third solution would not be too complicated (just some effort, to adapt all build tools for all packaging formats). However, it isn't sufficient to just provide debug plugins, rather it must also be ensured that a debug server loads a debug plugin (and an optimized server loads an optimized plugin). So we must only proceed on the "add debug plugins" route (packaging) if previously 1) the two alternatives (dedicated objects, stable interface) have been rejected for some valid reasons, 2) a new directory structure (or file naming) has been decided that clearly separates debug and optimized plugins, 3) the task to modify all affected code and scripts is realized and checked to be doable. Because of this complexity, I consider it extremely unlikely this can be solved for milestone 2, and I see a significant development risk. Setting Lead to "Plugins"
[1 Mar 2010 22:41]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/101974 3006 Mats Kindahl 2010-03-01 Bug #49022: Plugins included into bin release cannot be installed on debug version of server When starting a server using mysqld_safe with the option --mysqld-version=debug, the debug version of the server is started. When attempting to load a plugin that is not debug-built, the load will fail because a non-debug built plugin use, e.g., fastmutex while the debug-built server uses safemutex. To solve this problem, mysqld_safe will set the plugin-dir to use a directory where debug-built plugins are placed. This patch sets the plugin dir to <plugin-dir>/whatever, when mysqld-version=whatever, and just <plugin-dir> otherwise.
[4 Mar 2010 21:42]
Joerg Bruehe
Approved. I do not see any faults with this patch. However, it introduces a new option to "mysqld_safe", so this needs to be documented. By itself, it does not provide the debug plugin, for this we still need a build tool change (to put the plugin from the debug build into the package). I hope to do that change soon.
[8 Mar 2010 14:35]
Mats Kindahl
Hi Joerg, It actually does not introduce a new option to mysqld_safe, it just modifies it before passing it to mysqld (which of course should be documented). Since mysqld_safe passes any options it does not recognize to mysqld, passing --plugin-load to mysqld_safe before the patch works, after the patch, it is still passed to mysqld, but modified to handle the actual location of the plugins based on the mysqld-version started.
[9 Mar 2010 21:35]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/102783 3117 Joerg Bruehe 2010-03-09 Part of the fixes for bug#49022 Plugins included into bin release cannot be installed on debug version of server IF the build process was split into separate "debug" and "optimized" builds AND the plugin files of the debug build got copied into "plugin/debug/" (both is done for MySQL release builds starting from 5.5.3), THEN these debug plugin files are to be included in the final binary package. This change deals with the inclusion only, the other parts are done in different changesets. @ plugin/Makefile.am The "install" rule must not assume that "debug" files are present, this depends on the preceding steps in the build. If they are present, we cannot copy them by simply using "libtool install" (as we could do during an ordinary build) because they are already kept in "plugin/debug/". When doing the copy to the destination hierarchy, we must ensure that symlinks are not expanded but rather copied as symlinks. "cp -d" is specific to GNU, the portable way would use "cp -P", but that is not fully specified with recursion. So we fall back on "tar c | tar x", which is known to keep symlinks unchanged (and un-expanded).
[10 Mar 2010 9:49]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/102835 3117 Joerg Bruehe 2010-03-10 Part of the fixes for bug#49022 Plugins included into bin release cannot be installed on debug version of server IF the build process was split into separate "debug" and "optimized" builds AND the plugin files of the debug build got copied into "plugin/debug/" (both is done for MySQL release builds starting from 5.5.3), THEN these debug plugin files are to be included in the final binary package. This change deals with the inclusion only, the other parts are done in different changesets. @ plugin/Makefile.am The "install" rule must not assume that "debug" files are present, this depends on the preceding steps in the build. If they are present, we cannot copy them by simply using "libtool install" (as we could do during an ordinary build) because they are already kept in "plugin/debug/". When doing the copy to the destination hierarchy, we must ensure that symlinks are not expanded but rather copied as symlinks. "cp -d" is specific to GNU, the portable way would use "cp -P", but that is not fully specified with recursion. So we fall back on "tar c | tar x", which is known to keep symlinks unchanged (and un-expanded). Using "$(TAR)" is just a precaution in case of weird path settings or other portability issues.
[10 Mar 2010 11:15]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/102861 3123 Joerg Bruehe 2010-03-10 [merge] Automerge part of the fixes for bug#49022 into "next-mr-bugfixing".
[10 Mar 2010 13:37]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/102887 3118 Joerg Bruehe 2010-03-10 Part of the fixes for bug#49022 Plugins included into bin release cannot be installed on debug version of server Ensure that the plugin files of the debug build get into the optimized tree, so that they find their way into the final RPMs. @ support-files/mysql.spec.sh Use "make install" in the debug build tree to get the debug plugin files to a temporary location, then move them to "plugin/debug/" in the optimized build tree so that the install hook in "plugin/Makefile.am" forwards them into the final installation. This fixes bug#49022 for the generic RPMs. Unrelated: The plugin objects were listed twice in the file list, do a cleanup.
[10 Mar 2010 13:38]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100310133305-0jdlngbtrkoqzckh) (version source revid:alik@sun.com-20100310133204-g47qu3vqyquj172z) (merge vers: 6.0.14-alpha) (pib:16)
[10 Mar 2010 13:38]
Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100310132634-zpyjzn346sgrm59u) (version source revid:alik@sun.com-20100310132634-zpyjzn346sgrm59u) (merge vers: 5.5.3-m3) (pib:16)
[10 Mar 2010 13:39]
Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100310132718-l0tegumhbg8umgjd) (version source revid:alik@sun.com-20100310132252-kpi29r22kjtl493x) (pib:16)
[11 Mar 2010 15:40]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/103025 3119 Joerg Bruehe 2010-03-11 Fix for the bug#49022 patch, detected by further testing.
[12 Mar 2010 9:59]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/103059 3128 Joerg Bruehe 2010-03-12 [merge] Automerge the bug#49022 fix correction into next-mr-bugfixing.
[12 Mar 2010 18:11]
Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100312180926-0emfjrj8e9xnvl8h) (version source revid:alik@sun.com-20100312180447-2r0ak22y13s05134) (merge vers: 6.0.14-alpha) (pib:16)
[12 Mar 2010 18:11]
Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100312180838-rk60kn38g0qwt78n) (version source revid:alik@sun.com-20100312180435-wk7nvsbfntfus5bu) (pib:16)
[12 Mar 2010 18:22]
Bugs System
Pushed into 5.5.3-m3 (revid:alik@sun.com-20100312181131-0b7v8r2htpd9jz2a) (version source revid:alik@sun.com-20100312181131-0b7v8r2htpd9jz2a) (merge vers: 5.5.3-m3) (pib:16)
[23 Aug 2010 13:16]
Paul DuBois
Noted in 5.5.3, 6.0.14 changelogs. Plugins in a binary release could not be installed into a debug version of the server.
[24 Nov 2010 14:23]
MySQL Verification Team
See bug: http://bugs.mysql.com/bug.php?id=58460.
[27 Jan 2011 12:07]
Luis Soares
See also: BUG#59148.
