| Bug #56817 | mysqlhotcopy tests fail when DBI and DBD-mysql perl modules are not found | ||
|---|---|---|---|
| Submitted: | 16 Sep 2010 12:04 | Modified: | 9 Feb 2011 19:23 |
| Reporter: | John Embretsen | Email Updates: | |
| Status: | Closed | Impact on me: | |
| Category: | MySQL Server: Tests | Severity: | S7 (Test Cases) |
| Version: | 5.1, 5.5, 5.6 | OS: | Any |
| Assigned to: | John Embretsen | CPU Architecture: | Any |
| Tags: | disabled | ||
[17 Sep 2010 15:03]
John Embretsen
My investigations so far have confirmed the following: - The main issue described in the synopsis applies to 5.1 as well. - In addition, 5.1 suffers from a deficiency in MTR when it comes to looking up the location of mysqlhotcopy. So the test fails also for this reason in 5.1. - A fix for the MTR deficiency was pushed to "5.5" (trunk-bugfixing etc.) 2010-08-26 by with no reference to any bug report or worklog, e.g. revid holyfoot@mysql.com-20100825135522-8wrjjmvh2u2t0qo6. This fix was NOT pushed to 5.1 - this needs to be done. - The fix was disabled and made experimental by different persons at different times, citing Bug#54129. In short: It's a mess. It seems like it _is_ possible (by taking some detours) to check if the required perl modules are available and skip the test if not. I plan to look into fixing this early next week.
[21 Sep 2010 13:58]
John Embretsen
Opened Bug#56895 for handling the changes mentioned above that are a) present in 5.5, trunk and next-mr, but not present in 5.1 b) needed in 5.1 c) not related to missing Perl modules.
[22 Sep 2010 9:19]
John Embretsen
Helper perl script that can be used for checking support for DBI and DBD::mysql modules.
Attachment: checkDBI_DBD-mysql.pl (application/octet-stream, text), 2.55 KiB.
[23 Sep 2010 11:08]
John Embretsen
Waiting for fix of Bug#56895 before committing patch.
[26 Jan 2011 13:36]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/129665 3554 John H. Embretsen 2011-01-26 Fix for Bug#56817 - mysqlhotcopy tests fail when DBI and DBD-mysql perl modules are not found This patch implements a check for such modules at runtime. If modules are not found or unable to load, the test is skipped with the following message: [ skipped ] Test needs Perl modules DBI and DBD::mysql Checks are done via a helper Perl script which looks for the module in a runtime environment that is as similar to that of the mysqlhotcopy script as possible. The helper script tells mysql-test about the result by updating certain values in a test table created for the purpose. See comments in added files (have_dbi_dbd-mysql.inc and checkDBI_DBD-mysql.pl) for details. The patch also removes the mysqlhotcopy tests from the list of disabled tests.
[27 Jan 2011 10:34]
Horst Hunger
Please update disabled.def.
[27 Jan 2011 10:50]
Horst Hunger
Inserted the patch into mysql-5.1 and run the test with and without perl modules. Ready to push.
[27 Jan 2011 11:01]
John Embretsen
Thank you Horst. I just got a suggestion from Bjorn for an alternative solution, so I will explore that before going further. I will post the results later.
[28 Jan 2011 9:26]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/129830 3555 John H. Embretsen 2011-01-28 Fix for Bug#56817 - mysqlhotcopy tests fail when DBI and DBD-mysql perl modules are not found Updated patch based on feedback on the previous commit for this bug. This patch implements a check for such modules at runtime. If modules are not found or unable to load, the test is skipped with the following message: [ skipped ] Test needs Perl modules DBI and DBD::mysql Checks are done via a helper Perl script which looks for the module in a runtime environment that is as similar to that of the mysqlhotcopy script as possible. The helper script tells mysql-test about the result by writing information to a temporary file that is later read by mysql-test. See comments in added files (have_dbi_dbd-mysql.inc and checkDBI_DBD-mysql.pl) for details. The patch also removes the mysqlhotcopy tests from the list of disabled tests.
[28 Jan 2011 9:36]
John Embretsen
Committed new patch (http://lists.mysql.com/commits/129830) which supersedes the previous patch, based on a suggestion from Bjørn to use a temporary file instead of database tables for information transport from Perl to mysql-test. This is a slight simplification compared to the previous patch. Horst, sorry to create this extra work for you, but can you please formally check that the updated patch looks OK? It is the same as the previous one except that it writes to and reads from a file instead of connecting to the test database from Perl. Works well in my test environments. Bjørn, can you check that this patch follows more or less your suggestions wrt. sending information between Perl and mysql-test? Thanks!
[28 Jan 2011 10:03]
Bjørn Munch
I'm OK with the use of a file to communicate from Perl. Just one thing: in the example in the large comment block, you refer to $perlChecker instead if $resultFile in two places (I think it's two).
[28 Jan 2011 10:55]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/129839 3555 John H. Embretsen 2011-01-28 Fix for Bug#56817 - mysqlhotcopy tests fail when DBI and DBD-mysql perl modules are not found Second updated patch based on feedback on the previous commit for this bug. Fixed example in comments of checkDBI_DBD-mysql.pl and made it clear in the include file check that Windows is not supported at this time (mysqlhotcopy test does not support Windows either). This patch implements a check for such modules at runtime. If modules are not found or unable to load, the test is skipped with the following message: [ skipped ] Test needs Perl modules DBI and DBD::mysql Checks are done via a helper Perl script which looks for the module in a runtime environment that is as similar to that of the mysqlhotcopy script as possible. The helper script tells mysql-test about the result by writing information to a temporary file that is later read by mysql-test. See comments in added files (have_dbi_dbd-mysql.inc and checkDBI_DBD-mysql.pl) for details. The patch also removes the mysqlhotcopy tests from the list of disabled tests.
[28 Jan 2011 15:06]
Horst Hunger
2. Review: OK to push.
[29 Jan 2011 11:38]
Bugs System
A patch for this bug has been committed. After review, it may be pushed to the relevant source trees for release in the next version. You can access the patch from: http://lists.mysql.com/commits/129937 3558 John H. Embretsen 2011-01-29 Fix for Bug#56817 - mysqlhotcopy tests fail when DBI and DBD-mysql perl modules are not found Third updated patch - this version also includes copyright notice in added Perl script. This patch implements a check for such modules at runtime. If modules are not found or unable to load, the test is skipped with the following message: [ skipped ] Test needs Perl modules DBI and DBD::mysql Checks are done via a helper Perl script which looks for the module in a runtime environment that is as similar to that of the mysqlhotcopy script as possible (thus not intended for Windows environments at this time). The helper script tells mysql-test about the result by writing information to a temporary file that is later read by mysql-test. See comments in added files (have_dbi_dbd-mysql.inc and checkDBI_DBD-mysql.pl) for details. The patch also removes the mysqlhotcopy tests from the list of disabled tests.
[29 Jan 2011 12:18]
Bugs System
Pushed into mysql-trunk 5.6.2 (revid:john.embretsen@oracle.com-20110129121556-1162ss59b7ivqn2z) (version source revid:john.embretsen@oracle.com-20110129121556-1162ss59b7ivqn2z) (merge vers: 5.6.2) (pib:24)
[29 Jan 2011 12:18]
Bugs System
Pushed into mysql-5.5 5.5.10 (revid:john.embretsen@oracle.com-20110129114955-yhdakwd57ew4bnmo) (version source revid:john.embretsen@oracle.com-20110129114955-yhdakwd57ew4bnmo) (merge vers: 5.5.10) (pib:24)
[29 Jan 2011 12:19]
Bugs System
Pushed into mysql-5.1 5.1.56 (revid:john.embretsen@oracle.com-20110129113836-dfqep5evpzbl1pqq) (version source revid:john.embretsen@oracle.com-20110129113836-dfqep5evpzbl1pqq) (merge vers: 5.1.56) (pib:24)
[29 Jan 2011 14:24]
John Embretsen
Patch pushed to 5.1, 5.5 and trunk on 2011-01-29. mysqlhotcopy tests re-enabled in all branches. Test is now skipped on Solaris platforms in Pushbuild due to Perl modules DBI and DBD::mysql not being installed for the Perl installation at /usr/bin/perl, which the mysqlhotcopy script points to by default. For most generic Linux platforms that perl installation has the required modules, or in some cases Pushbuild itself builds and bundles DBD::mysql with each build, so the test is run there (and seems to pass).
[9 Feb 2011 19:23]
Paul DuBois
Changes to test suite. No changelog entry needed.
Description: There are two mysqlhotcopy tests in the main test suite: mysql-test/t/mysqlhotcopy_archive.test mysql-test/t/mysqlhotcopy_myisam.test Currently, both tests fail on some platforms in Pushbuild with what looks like environmental issues related to missing or incomplete Perl module installations on specific machines. The tests test a perl script (mysqlhotcopy), which requires the Perl modules DBI and DBD::mysql. Example symptom (from mysql-trunk, 2010-09-14, Linux x86_64 gcov): --------------------- main.mysqlhotcopy_myisam [ exp-fail ] Test ended at 2010-09-13 16:32:08 CURRENT_TEST: main.mysqlhotcopy_myisam install_driver(mysql) failed: Can't load '/usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/auto/DBD/mysql/mysql.so' for module DBD::mysql: libmysqlclient.so.15: cannot open shared object file: No such file or directory at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230. at (eval 7) line 3 Compilation failed in require at (eval 7) line 3. Perhaps a required shared library or dll isn't installed where expected at /export/home2/pb2/build/sb_1-None-1284385899.28/mysql-trunk-gcov/scripts/mysqlhotcopy line 180 mysqltest: In included file "./include/mysqlhotcopy.inc": At line 32: command "$MYSQLHOTCOPY --quiet -S $MASTER_MYSOCK -u root hotcopy_test hotcopy_save" failed Output from before failure: exec of '/export/home2/pb2/build/sb_1-None-1284385899.28/mysql-trunk-gcov/scripts/mysqlhotcopy --quiet -S /export/home2/pb2/build/sb_1-None-1284385899.28/mysql-trunk-gcov/mysql-test/var/tmp/mysqld.1.sock -u root hotcopy_test hotcopy_save' failed, error: 512, status: 2, errno: 0 --------------------- This bug can be considered a Pushbuild bug (missing test libraries). However, it would also help reducing test failure noise by making the tests more robust against these environmental failures, i.e. skip test if requirements are not met. Note that in 5.1 the tests currently seem to fail (had they not been disabled) due to "missing mysqlhotcopy tool". It is assumed that they would have failed in the same way as in 5.5 had the tool been found. A related issue is that the mysqlhotcopy tests cannot be run (and pass) on a system with Perl (+ modules) installed in a location that is not /usr/bin/ (on Unix/Linux), due to the "shebang" construct in the mysqlhotcopy script. How to repeat: On a machine without DBI or DBD::mysql perl modules installed or in PATH (but with some kind of Perl installation in order to run MTR), do: perl mysql-test-run.pl --enable-disabled mysqlhotcopy_myisam Suggested fix: If possible, I suggest fixing this in two stages: 1. Make the mysqlhotcopy tests more robust: - Skip the tests automatically if the required Perl modules are not found, or for some reason cannot be loaded properly. There could be an include file checking for this. - This will make the tests run as intended if the modules are found (with the exception of 5.1 as mentioned above. Otherwise the tests will be skipped with a specific reason. 2. Make the required perl modules available on all test hosts in Pushbuild.