Bug #39774 mysql-test-run's remove_file can't use wildcards, this should be documented
Submitted: 1 Oct 2008 13:26 Modified: 17 Jun 2010 19:42
Reporter: Patrick Crews Email Updates:
Status: Closed Impact on me:
None 
Category:Tools: MTR / mysql-test-run Severity:S3 (Non-critical)
Version: OS:Any
Assigned to: Bjørn Munch CPU Architecture:Any
Tags: Docs, mysql-test-run, remove_file, wildcard
Triage: Triaged: D3 (Medium)

[1 Oct 2008 13:26] Patrick Crews
Description:
mysql-test-run's remove_file command is fails when passed wildcards.  This behavior should be documented or altered.

While it could be argued that remove_file is behaving as documented - returns error 1 if the file does not exist, the fact that this is meant to replace the use of the OS's rm or del utilities implies that wildcards should be accepted.

How to repeat:
Place the following lines in a .test file and execute the following command:

./mtr --record <name_of_test_file>

#Test file content
--exec echo a > a.foo
--exec echo b > b.foo

--remove_file *.foo

You will see this error:
mysqltest: At line 4: command "remove_file" failed with error 1
 and the .foo files will still be present.

Suggested fix:
Either adjust the remove_file code to accept wildcards or make a note in documentation that wildcards (currently) do not work.

I feel that since this is mean to be a substitute for an OS's rm / del commands, the use of wildcards is implied.  We need to be explicit about the limitations of this function.
[1 Oct 2008 13:45] Patrick Crews
After consultation with Docs team, re-categorizing it to Server:Tests until we determine if this should be handled via changing mtr or just documenting the existing behavior
[30 Oct 2008 9:16] Magnus Blåudd
The remove_file in mysqltest has intentionally been written this way to make it very explicit _exactly_ which files are delete.

Please add to the documentation.
[30 Oct 2008 15:35] Paul Dubois
Recategorizing as Server: Documentation, assigning to myself.
[7 Nov 2008 17:07] Paul Dubois
Thank you for your bug report. This issue has been addressed in the documentation. The updated documentation will appear on our website shortly, and will be included in the next release of the relevant products.

Filename argument must evaluate to literal filename, not filename pattern.

Also true for chmod_file.
[13 May 2009 11:22] Philip Stoev
Magnus,

I am afraid that we will need the wildcard remove file thingy in order to fix sporadic failures in our tests. What is your rationale to require remove-file to list the file explicitly?

If you want to keep remove-file as it is, we would need a remove-file-wildcard that will take wildcards (but not recursive delete). If you are concerned about rm -rf / , we should make remove-file-wildcard operate only inside the vardir.

Please let me know.

Philip Stoev
[2 Oct 2009 15:51] Mattias Jonsson
I need this to be able to test error handling in partitioning (which can currently leave a lot of temporary files in a specific test case...) So I will implement remove_files which will take directory as first argument and optionally filenamepattern as a second argument. If the second argument is empty it will remove all files within that directory, otherwise it will only remove matching files. Just like list_files works, but instead of listing them, it will remove them.
[6 Oct 2009 9:14] Magnus Blåudd
Please notify Bjorn before coding.
[6 Oct 2009 10:27] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/85855

3147 Mattias Jonsson	2009-10-06
      Bug#39774: mysql-test-run's remove_file can't use wildcards, this should be documented
      
      Needed to be able to clean up in an efficient way with wildcards.
      
      Added remove_files_wildcard that allows to remove multiple files
      at once.
      
      Removes only regular files (i.e. no directories and is not recursive).
      Works like list_files.
      
      Note: the pattern is match within the string, i.e. not matching start/end,
      so 'file*.txt' also matches list_file.txt for example. 
     @ client/mysqltest.cc
        Bug#39774: mysql-test-run's remove_file can't use wildcards, this should be documented
        
        Added remove_files_wildcard that allows to remove multiple files
        at once.
     @ mysql-test/r/mysqltest.result
        Bug#39774: mysql-test-run's remove_file can't use wildcards, this should be documented
        
        Updated test result
     @ mysql-test/t/mysqltest.test
        Bug#39774: mysql-test-run's remove_file can't use wildcards, this should be documented
        
        Added tests for remove_files_wildcard
[2 Feb 2010 14:25] Bjørn Munch
This worked on Solaris but when I applied the patch in Windows and tried to run the mysqltest.test, I got

command "remove_files_wildcard" failed with error  -858993460

This is from the first test using file?.txt

Any idea?
[2 Feb 2010 14:33] Bjørn Munch
BTW, that error code it reports is CCCCCCCC in hex
[3 Feb 2010 12:28] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/99049

2886 Bjorn Munch	2010-02-03
      Bug #39774 mysql-test-run's remove_file can't use wildcards, this should be documented
      Added remove_files_wildcard that allows to remove multiple files at once.
      This is a port of previous patch to Windows, plus resolving a merge 
      conflict with current 5.1-mtr branch.
[4 Feb 2010 9:34] Magnus Blåudd
Looks ok, awaiting patch that fixes Mattias comments.
[4 Feb 2010 9:56] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/99223

2886 Bjorn Munch	2010-02-04
      Bug #39774 mysql-test-run's remove_file can't use wildcards, this should be documented
      Added remove_files_wildcard that allows to remove multiple files at once.
      This is a port of original patch to Windows.
[4 Feb 2010 11:00] Magnus Blåudd
Don't leave the non working code. Since we decided to change to "remove everything that is not a dir" it's no longer relevant since IS_DIR more or lees means !IS_REG

Checked the implementation of my_dir and yes, it does not set the IS_REG bit in the windows implementation.
[4 Feb 2010 11:12] Bjørn Munch
I have filed Bug #50877 for the missing flag in my_dir().

As far as I can see, the only difference is that we won't skip "special files" like character or block devices or pipes, but I don't see this as a big problem.

I will follow Bug #50877 (since I'm the reporter) and can ask whoever picks it up to also undo my workaround.
[4 Feb 2010 12:16] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/99266

2887 Bjorn Munch	2010-02-04
      Bug #39774 mysql-test-run's remove_file can't use wildcards, this should be documented
      Added remove_files_wildcard that allows to remove multiple files at once.
      This is a port of original patch to Windows.
[4 Feb 2010 12:19] Bjørn Munch
Pushed to 5.1-mtr, trunk-mtr, next-mr-mtr, 6.0-codebase-mtr.

Sorry about the extra commit; silly bzr mistake
[20 Feb 2010 9:31] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20100220092844-jh07ayojyxn8uh9p) (version source revid:bjorn.munch@sun.com-20100207141526-5m83mmrz9ad2ouh1) (merge vers: 6.0.14-alpha) (pib:16)
[20 Feb 2010 9:33] Bugs System
Pushed into 5.5.3-m2 (revid:alik@sun.com-20100220092622-wvhh1vfy5tjq4mhu) (version source revid:bjorn.munch@sun.com-20100207113713-ng22k3dbavz6yohg) (merge vers: 5.5.2-m2) (pib:16)
[20 Feb 2010 9:36] Bugs System
Pushed into mysql-next-mr (revid:alik@sun.com-20100220092730-paoo5t9gcajs5dq8) (version source revid:bjorn.munch@sun.com-20100207114124-ath7uwrv9cjhpeb6) (pib:16)
[20 Feb 2010 18:35] Paul Dubois
No changelog entry needed.
[22 Feb 2010 20:30] Paul Dubois
Noted in 5.5.3, 6.0.14 changelogs.

mysqltest has a new remove_files_wildcard command that removes files
matching a pattern from a directory.  

Setting report to Need Merge pending push to 5.1.x, Celosia.
[1 Mar 2010 8:46] Bugs System
Pushed into 5.1.45 (revid:joro@sun.com-20100301083827-xnimmrjg6bh33o1o) (version source revid:azundris@mysql.com-20100222175719-viuh0f3gdsrkgv0r) (merge vers: 5.1.45) (pib:16)
[1 Mar 2010 16:24] Paul Dubois
Noted in 5.1.45 changelog.
[9 Jun 2010 14:12] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110637

3103 Martin Skold	2010-06-09 [merge]
      Merged in 5.1.45
      added:
        mysql-test/include/not_binlog_format_row.inc
        mysql-test/r/bug39022.result
        mysql-test/r/no_binlog.result
        mysql-test/r/partition_debug_sync.result
        mysql-test/std_data/bug48449.frm
        mysql-test/suite/rpl/r/rpl_slow_query_log.result
        mysql-test/suite/rpl/t/rpl_slow_query_log-slave.opt
        mysql-test/suite/rpl/t/rpl_slow_query_log.test
        mysql-test/t/bug39022.test
        mysql-test/t/no_binlog.test
        mysql-test/t/partition_debug_sync.test
      renamed:
        mysql-test/r/variables+c.result => mysql-test/r/variables_community.result
        mysql-test/t/variables+c.test => mysql-test/t/variables_community.test
      modified:
        client/mysql.cc
        client/mysql_upgrade.c
        client/mysqladmin.cc
        client/mysqlbinlog.cc
        client/mysqlcheck.c
        client/mysqldump.c
        client/mysqlimport.c
        client/mysqlshow.c
        client/mysqlslap.c
        client/mysqltest.cc
        cmd-line-utils/readline/rlmbutil.h
        cmd-line-utils/readline/text.c
        configure.in
        extra/yassl/include/yassl_error.hpp
        extra/yassl/src/ssl.cpp
        extra/yassl/src/yassl_error.cpp
        mysql-test/include/mtr_warnings.sql
        mysql-test/lib/My/ConfigFactory.pm
        mysql-test/lib/My/SafeProcess.pm
        mysql-test/lib/My/SafeProcess/safe_process_win.cc
        mysql-test/lib/mtr_cases.pm
        mysql-test/lib/mtr_gprof.pl
        mysql-test/lib/mtr_misc.pl
        mysql-test/lib/mtr_report.pm
        mysql-test/lib/mtr_stress.pl
        mysql-test/lib/v1/mtr_stress.pl
        mysql-test/lib/v1/mysql-test-run.pl
        mysql-test/mysql-test-run.pl
        mysql-test/r/archive.result
        mysql-test/r/backup.result
        mysql-test/r/bigint.result
        mysql-test/r/csv.result
        mysql-test/r/default.result
        mysql-test/r/delete.result
        mysql-test/r/fulltext.result
        mysql-test/r/func_gconcat.result
        mysql-test/r/func_time.result
        mysql-test/r/group_by.result
        mysql-test/r/group_min_max.result
        mysql-test/r/having.result
        mysql-test/r/innodb-autoinc.result
        mysql-test/r/innodb_mysql.result
        mysql-test/r/join.result
        mysql-test/r/log_state.result
        mysql-test/r/multi_update.result
        mysql-test/r/myisam.result
        mysql-test/r/mysqltest.result
        mysql-test/r/show_check.result
        mysql-test/r/sp-bugs.result
        mysql-test/r/sp-error.result
        mysql-test/r/sp.result
        mysql-test/r/sp_notembedded.result
        mysql-test/r/sp_trans.result
        mysql-test/r/subselect.result
        mysql-test/r/type_bit.result
        mysql-test/r/type_blob.result
        mysql-test/r/type_timestamp.result
        mysql-test/r/view.result
        mysql-test/r/view_grant.result
        mysql-test/r/warnings.result
        mysql-test/suite/rpl/r/rpl_sp.result
        mysql-test/suite/rpl/t/rpl_loaddata_symlink.test
        mysql-test/suite/sys_vars/r/log_basic.result
        mysql-test/suite/sys_vars/r/log_bin_trust_routine_creators_basic.result
        mysql-test/suite/sys_vars/r/slow_query_log_func.result
        mysql-test/suite/sys_vars/t/slow_query_log_func.test
        mysql-test/t/archive.test
        mysql-test/t/bigint.test
        mysql-test/t/csv.test
        mysql-test/t/delete.test
        mysql-test/t/fulltext.test
        mysql-test/t/func_gconcat.test
        mysql-test/t/group_by.test
        mysql-test/t/group_min_max.test
        mysql-test/t/having.test
        mysql-test/t/innodb-autoinc.test
        mysql-test/t/innodb_mysql.test
        mysql-test/t/join.test
        mysql-test/t/multi_update.test
        mysql-test/t/myisam.test
        mysql-test/t/mysqltest.test
        mysql-test/t/sp-bugs.test
        mysql-test/t/sp_notembedded.test
        mysql-test/t/subselect.test
        mysql-test/t/type_bit.test
        mysql-test/t/view.test
        mysql-test/t/view_grant.test
        mysys/default.c
        scripts/mysqld_multi.sh
        server-tools/instance-manager/options.cc
        sql/field.cc
        sql/ha_partition.cc
        sql/item.cc
        sql/item.h
        sql/item_cmpfunc.h
        sql/item_row.cc
        sql/item_sum.cc
        sql/item_timefunc.cc
        sql/log.cc
        sql/log_event.cc
        sql/log_event_old.cc
        sql/mysql_priv.h
        sql/mysqld.cc
        sql/opt_range.cc
        sql/opt_sum.cc
        sql/share/errmsg.txt
        sql/sp.cc
        sql/sp_cache.cc
        sql/sp_head.cc
        sql/sql_base.cc
        sql/sql_class.cc
        sql/sql_class.h
        sql/sql_lex.cc
        sql/sql_lex.h
        sql/sql_parse.cc
        sql/sql_repl.cc
        sql/sql_select.cc
        sql/sql_select.h
        sql/sql_table.cc
        sql/sql_trigger.cc
        sql/sql_update.cc
        sql/sql_view.cc
        sql/table.cc
        sql/table.h
        storage/archive/ha_archive.cc
        storage/csv/ha_tina.cc
        storage/innobase/buf/buf0buf.c
        storage/innobase/buf/buf0rea.c
        storage/innobase/handler/ha_innodb.cc
        storage/innobase/include/buf0rea.h
        storage/innobase/lock/lock0lock.c
        storage/innobase/os/os0file.c
        storage/myisam/ft_boolean_search.c
        storage/myisam/mi_dynrec.c
        storage/myisam/mi_locking.c
        support-files/mysql.spec.sh
[17 Jun 2010 12:00] Bugs System
Pushed into 5.1.47-ndb-7.0.16 (revid:martin.skold@mysql.com-20100617114014-bva0dy24yyd67697) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)
[17 Jun 2010 12:41] Bugs System
Pushed into 5.1.47-ndb-6.2.19 (revid:martin.skold@mysql.com-20100617115448-idrbic6gbki37h1c) (version source revid:martin.skold@mysql.com-20100609140708-52rvuyq4q500sxkq) (merge vers: 5.1.45-ndb-6.2.19) (pib:16)
[17 Jun 2010 13:27] Bugs System
Pushed into 5.1.47-ndb-6.3.35 (revid:martin.skold@mysql.com-20100617114611-61aqbb52j752y116) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)