Bug #20037 mysqltest requires cygwin on windows(part 1, new mysqltest commands)
Submitted: 24 May 2006 8:54 Modified: 7 Aug 2007 19:43
Reporter: Magnus Blåudd Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Tests Severity:S3 (Non-critical)
Version: OS:Windows (windows)
Assigned to: Magnus Blåudd CPU Architecture:Any

[24 May 2006 8:54] Magnus Blåudd
Description:
When running mysqltest on Windows it requires Cygwin. It should be possible to run without it.

A plan for how ho to do this is as follows:

>below is my suggestion on how to get rid of our dependcy on cygwin. 
>> Comments are appreicated. If I don't here anything from you 
>> within this week, I will proceed with the below suggestion.
>> 
>> 
>> Best regards
>> Magnus
>> 
>> ** Current commands in mysqltest:
>> 
>> --exec <command>
>> Currently used to execute some system dependent commands. 
>> Typical example
>> would be to 'echo' something to mysql('echo' does not work on windows
>> the same way as echo on linux) or call 'rm', 'mv'.
>> There is also one test where exec is used to execute "grep" in order
>> to remove some parts in a .TRG file.
>> 
>> It should be changed  used _only_ to execute our own binaries
>> like mysql, mysqltest, mysqldump, mysql etc. Maybe we can
>> enforce this by checking that the program being executed is in
>> the list of allowed programs to execute. Will append the output
>> from these commands to the result file.
>> 
>> In order to avoid rewiring the testcases too much, we might want to
>> allow  "exec" of commands on the
>> form "--exec echo "something" | <MYSQL_PROGRAM>, and handle it
>> internally in mysql, but it's probably easier to rewrite the testcase
>> to use an intermediate file.
>> 
>> It's common to exec something and pipe stderr to stdout like 
>> "2>&1", if
>> we can't get that to work on windows we should rewrite it on the fly
>> before executing.
>> 
>> 
>> --system <command>
>> Currently used to execute some system commands. Not used that 
>> frequently.
>> When used it will call 'rm', 'rm -f', 'rmdir', 'grep', 'mv', 'echo',
>> 'chmod', 'cp', 'cat', 'test -s'.
>> To be removed?
>> 
>> 
>> ** New commands to be added to mysqltest:
>> 
>> --rm <filename/directoryname>
>> Removes a file or directory. Replaces 'rm', 'rm -f' and 'rmdir'
>> 
>> --cp <from> <to>
>> Copy a file. Replaces 'cp'.
>> 
>> --mv <from> <to>
>> Move a file. Replaces 'mv'. Actually just a "cp" followed by a "rm".
>> 
>> 
>> --file_exists <filename>
>> Check for a file existence. Replace 'test -s'
>> Can be combined with an expected "--error 1" to detect that a 
>> file does not
>> exist
>> 
>> --perl <scriptname> <arg1> .. <argn>
>> Call a perl script and include it's output in the result log
>> (unless --disable_result_log has been used)
>> 
>> --write <filename> <what to write>
>> Writes the <what to write> to the file <filename>
>> 
>> --chmod <mod> <filename>
>> I am not sure if this one should be added.
>> Change the file permissions of the file. Should support only the mode
>> flags "-r", "+r", "-w" and "+w" as these are the only ones
>> that exist on windows. The "-r" will return the same error as 
>> if the file
>> doesn't exist at all on windows.
>> 
>> 
>> * Not to be implemented
>> cat
>> Only used in one place "cat /dev/null > file" and that could 
>> be replace
>> by a small perl script. Or maybe just "write <filename> 0" would do?
>> 
>> grep
>> Only used in two places. Need to call perl script. There can 
>> be standard
>> script grep.pl, but I think the "search and replace" thing 
>> that is done 
>> is really just a "one liner" in perl.
>> 
>> 
>> 

How to repeat:
I created this mainly as a place holder to keep track of it.

Suggested fix:
See above.
[14 Jun 2006 15:31] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/7639
[31 Jul 2006 14:24] Magnus Blåudd
This is the first patch for this bug, which adds new commands to mysqltest.
http://lists.mysql.com/commits/9820

Next step is to remove the "cygwin hacks" from mysqltest and get the things that don't work after that running with the help of the new commands.
[6 Sep 2006 13:55] Chad MILLER
Even better, use the shell here-document convention for specifying the end.  Three letters "EOF" shouldn't be special.

--perl <<XYZ
print "asdf asdf \
EOF is legal here.\n";
XYZ

$ man 1 bash  #search for "Here Documents"
[2 Aug 2007 7:43] Magnus Blåudd
The new commands has been added to mysqltest and now needs to be documented. Please see http://lists.mysql.com/commits/9820 for an explanation (and examples) of what has been added.

Follow up bug reports have been written for remaining issues running mysql-test-run.pl on Windows without cygwin.
[2 Aug 2007 7:48] Magnus Blåudd
As an effect of these new commands all uses of "--exec rm <file>" should be banned from .test scripts. Instead we should use remove_file etc.

Please update/remove the section of http://dev.mysql.com/doc/mysqltest/en/mysqltest-commands.html that says:
"On Cygwin, the command is executed from cmd.exe, so commands such as rm cannot be executed with exec. Use system  instead.

--exec $MYSQL_DUMP --xml --skip-create test
--exec rm $MYSQLTEST_VARDIR/tmp/t1
exec $MYSQL_SHOW test -v -v;" 
to indicate this.
[2 Aug 2007 8:06] Magnus Blåudd
Tried to find out what version this has gone into. I know it was done in 5.1 and then backported to 4.1 and 5.0 to get the same version of mysqltest in all version . This was done in August 2006, can't see exactly what version that we had at that time.
[7 Aug 2007 7:28] Magnus Blåudd
> New commands:
>
> 1) copy_file from_file to_file
>
> Fails if to_file exists.

ok

> 2) write_file file_name
>    file-content
>    EOF

It's "write_file file_name [DELIMITER]", i.e you can optionally choose what your delimiter should be.

> Overwrites any existing file.
>
> Why the inconsistency between treatment of existing files for
> copy_file and write_file?

Yes, changed "write_file" to throw an error if the file exist. Makes sense.

2b) There is also "append_file <file_name> [<delimiter>]"
2c) There is also "cat_file <file_name>"

> 3) file_exists file_name
>
> 4) remove_file file_name
>
> 5) perl
>   code to execute
>   EOF
>
> Are the filename arguments to these commands subject to variable
> substitution?

yes, the arguments are substituted, but not the content between the command and EOF

> Can the EOF terminator for write_file and perl be changed?

yes, there is an optional argument two write_file, append_file and perl that takes the delimiter to use.

6) There is also "diff_files file_name1 file_name2"

7) There is also chmod_file <octal> <filename>

8) Both "skip" and "exit" is in the manual but not "die <message>"

9) As part of bug#29579 the command "send_quit" was added, it will send COM_QUIT to the server. Please leave that out of the manual for a little while - maybe we should change that to a more generic command that allows the sending of any such command to the server?

10) The syntax for "let" has been extended to allow the retrieval of individual values in a larger result set(this is especially suited for reading values from a SHOW query)

let $variable= query_get_value(<query to run>,<column name>,<row no>);

Ex.
CREATE TABLE t1(a int, b varchar(255), c datetime);
SHOW COLUMNS FROM t1;
+Field	Type	Null	Key	Default	Extra
+a	int(11)	YES		NULL	
+b	varchar(255)	YES		NULL	
+c	datetime	YES		NULL

let $value= query_get_value(SHOW COLUMNS FROM t1, Type, 1);
echo $value;
+int(11)

11) eval_result
    Unknown. 

I suggest deprecating this command and removing it from the manual, it's not used anywhere.
[7 Aug 2007 9:40] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/32187

ChangeSet@1.2506, 2007-08-07 11:40:03+02:00, msvensson@pilot.(none) +7 -0
  Bug#20037 mysqltest requires cygwin on windows(part 1, new mysqltest commands)
   - Update comments
   - Make "write_file" fail if file already exist
   - Remove temporary files created by test cases
[7 Aug 2007 19:43] Paul DuBois
The mysqltest manual now describes these commands:

append_file
cat_file
chmod_file
copy_file
die
diff_files
file_exists
perl
remove_file
write_file

and extended syntax for the let command.
[4 Sep 2007 17:12] Bugs System
Pushed into 5.1.23-beta
[4 Sep 2007 17:13] Bugs System
Pushed into 5.0.50