MySQL Bugs: #55310: RQG: New Server module sometimes unable to find MySQL binaries on Windows

Bug #55310	RQG: New Server module sometimes unable to find MySQL binaries on Windows
Submitted:	16 Jul 2010 8:00	Modified:	27 Jun 2013 12:42
Reporter:	John Embretsen	Email Updates:
Status:	Won't fix	Impact on me:	None
Category:	Tools: Random Query Generator	Severity:	S1 (Critical)
Version:	bzr-revno481	OS:	Windows (Server 2003)
Assigned to:	John Embretsen	CPU Architecture:	Any

Description:
On Windows, when using runall-new.pl to execute RQG tests instead of runall.pl (runall-new uses the new Server module to start/stop/manage the server instead of MTR), RQG sometimes fails to find the correct binaries, resulting in errors.

Feel free to open separate bugs for each issue reported below if necessary. Issue 1 is most critical, as it does not allow execution of the test at all.

Issue 1: mysqld.exe:
--------------------

Cannot find 'mysqld.exe' in 'G:/pb2/test/sb_1-2047673-1279238332.7/sql/Debug','G:/pb2/test/sb_1-2047673-1279238332.7/sql/RelWithDebugInfo','G:/pb2/test/sb_1-2047673-1279238332.7/sql/Release', at runall-new.pl line 244

Searching in the binary archive made available by Pushbuild reveals that mysqld.exe is indeed located in sql/Debug, so this looks mysterious.

Note that this does not happen every time runall-new.pl is used, but more often than not.

Issue 2: mysqldump.exe:
----------------------

(...)
Cannot find 'mysqldump.exe' in 'G:/pb2/test/sb_1-2048647-1279238072.58/mysql-5.1.49-win-x86-test/client/Debug','G:/pb2/test/sb_1-2048647-1279238072.58/mysql-5.1.49-win-x86-test/client/RelWithDebugInfo','G:/pb2/test/sb_1-2048647-1279238072.58/mysql-5.1.49-win-x86-test/client/Release', at runall-new.pl line 244

(Note: mysqldump is used sometimes near the end of a test run. Not all test runs uses it.)

Searching in the binary archive made available by Pushbuild reveals that mysqldump.exe is indeed located in client/Debug, so this looks mysterious.

Issue 3: core and binary upon crash:
------------------------------------

# 2010-07-15T17:28:37 Query: SELECT CONCAT (...) failed: 2013 Lost connection to MySQL server during query # 2010-07-15T17:28:37 Server crash reported at dsn dbi:mysql:host=127.0.0.1:port=19300:user=root:database=test 
# 2010-07-15T17:28:38 Killing periodic reporting process with pid -3836... 
# 2010-07-15T17:28:39 Server crash reported, initiating post-crash analysis... # 2010-07-15T17:28:39 The last 100 lines from G:\pb2\test\sb_1-2048647-1279206552.35\mysql-5.1.49-win-x86-test\vardirs\mysql.err :
100715 17:11:33  InnoDB: Started; log sequence number 0 44233
100715 17:11:33 [Warning] G:\pb2\test\sb_1-2048647-1279206552.35\mysql-5.1.49-win-x86-test\sql\Debug\mysqld.exe: unknown variable 'loose-lock-wait-timeout=1'
100715 17:11:33 [Note] Event Scheduler: Loaded 0 events
100715 17:11:33 [Note] G:\pb2\test\sb_1-2048647-1279206552.35\mysql-5.1.49-win-x86-test\sql\Debug\mysqld.exe: ready for connections.
Version: '5.1.49-debug'  socket: ''  port: 19300  Source distribution
# 2010-07-15T17:28:39 datadir is G:\pb2\test\sb_1-2048647-1279206552.35\mysql-5.1.49-win-x86-test\vardirs\data\ 
# 2010-07-15T17:28:39 binary is 
# 2010-07-15T17:28:39 bindir is 
# 2010-07-15T17:28:39 core is 
# 2010-07-15T17:28:39 
# 2010-07-15T17:28:39 Microsoft (R) Windows Debugger Version 6.9.0003.113 X86 
# 2010-07-15T17:28:39 Copyright (c) Microsoft Corporation. All rights reserved. # 2010-07-15T17:28:39 
# 2010-07-15T17:28:39 
# 2010-07-15T17:28:39 Loading Dump File [G:\pb2\test\sb_1-2048647-1279206552.35\mysql-5.1.49-win-x86-test\vardirs\data\mysqld.dmp] 
# 2010-07-15T17:28:39 Could not open dump file [G:\pb2\test\sb_1-2048647-1279206552.35\mysql-5.1.49-win-x86-test\vardirs\data\\mysqld.dmp], Win32 error 0n2 
# 2010-07-15T17:28:39 "The system cannot find the file specified." 
# 2010-07-15T17:28:39 Debuggee initialization failed, Win32 error 0n2 
# 2010-07-15T17:28:39 "The system cannot find the file specified." 
(...)
# 2010-07-15T17:30:38 Test completed with failure status 101. 
# 2010-07-15T17:30:38 GenTest exited with exit status 101 
# 2010-07-15T17:30:38 Stopping server on port 19300
DBI connect('host=127.0.0.1:port=19300:user=root:database=mysql','',...) failed: Can't connect to MySQL server on '127.0.0.1' (10061) at lib/DBServer/MySQL/MySQLd.pm line 459
Can't call method "func" on an undefined value at lib/DBServer/MySQL/MySQLd.pm line 414.
Perl exited with active threads:
	0 running and unjoined
	73 finished and unjoined
	1 running and detached

How to repeat:
Issues mentioned above occurred against MySQL server versions 5.5-m3 or newer. Actual test results may vary.

Below, a working MySQL binary installation is referred to as %CODE%.
The RQG can be obtained by doing "bzr branch lp:randgen".
Each multi-line command below should be executed as a single command.

Issue 1 (mysqld.exe):

perl runall-new.pl 
 --grammar=conf/partitioning/partitions-ddl.yy 
 --threads=1 
 --queries=100K 
 --reporters=Deadlock,ErrorLog,Backtrace 
 --duration=600 
 --basedir=%CODE% 
 --mysqld=--innodb 
 --mysqld=--log-output=file 

(Seen against branch mysql-trunk-bugfixing, Pushbuild test rqg_partition_ddl (and many others), July 15)

Issue 2 (mysqldump.exe):

perl runall-new.pl 
 --grammar=conf/runtime/metadata_stability.yy 
 --gendata=conf/runtime/metadata_stability.zz 
 --validator=SelectStability,QueryProperties
 --queries=1M 
 --duration=600 
 --reporters=Deadlock,ErrorLog,Backtrace 
 --basedir=%CODE% 
 --engine=Innodb 
 --mysqld=--innodb 
 --mysqld=--default-storage-engine=Innodb 
 --mysqld=--transaction-isolation=SERIALIZABLE 
 --mysqld=--innodb-flush-log-at-trx-commit=2 
 --mysqld=--loose-table-lock-wait-timeout=1 
 --mysqld=--innodb-lock-wait-timeout=1 
 --mysqld=--log-output=file
 --mysqld=--loose-skip-safemalloc

(Seen against branch mysql-trunk-bugfixing, Pushbuild test rqg_mdl_stability, July 15)

Issue 3 (core, binary at crash):

perl runall-new.pl 
 --threads=1 
 --queries=75K 
 --grammar=conf/optimizer/optimizer_subquery.yy 
 --duration=1200 
 --reporters=Deadlock,ErrorLog,Backtrace 
 --basedir=%CODE% 

(Seen against branch mysql-5.1-bugteam, Pushbuild test rqg_opt_subquery, July 15)

Issue 1 and 2 can probably be fixed by adding "bin" to binary search path on Windows (MySQL.pm in DBServer module). Official releases and most Pushbuild builds have mysqld and mysqldump in "bin". Not sure why this is not always an issue, though - maybe some builds do not use "bin".

Pushed likely fix for Issue 1 and 2 as revid john.embretsen@oracle.com-20100804091242-p2spp2x9yvhpzy0w 2010-08-04.
http://bazaar.launchpad.net/~randgen/randgen/rqg2/revision/503

Issue 3 needs to be investigated further.

John, you have typo in your fix (and it looks like typo was also before your fix). It is not RelWithDebugInfo, it is RelWithDebInfo directory.
 
It is build directory, it is not how it should be packaged.

I'm not sure what and how PB2 packs for RQG, if it used standard cmake-ZIP packaging, you would not have to look in the build directories.

If your intention is run RQG in build directories, then looking in RelWithDebinfo etc directories on windows "only" would be not correct either, when building on OSX with XCode, you have the same directory structure.

Thanks for this info, Wlad!

I have pushed a fix to lp:randgen changing "RelWithDebugInfo" to "RelWithDebInfo".

http://bazaar.launchpad.net/~randgen/randgen/rqg2/revision/514

It is our desire to make the RQG able to run against both build directories and released binaries, although this bug report goes more against the latter (assuming Pushbuild does something similar to release builds).

The RQG has as far as I know not been written with explicit support for Mac, although there is no clear reason at this time why it should not work as well there as on other platforms (it worked the last time I tried). I might do a couple of test drives on Mac and make sure it at least runs with some basic configurations. Thanks for letting me know about the directory layout with xcode - I was not aware of that.

Closing as "Won't fix", due to not being a priority. Internally we use other frameworks for server handling on Windows, and use the RQG as client only.