MySQL Bugs: #39774: mysql-test-run's remove_file can't use wildcards, this should be documented

Bug #39774	mysql-test-run's remove_file can't use wildcards, this should be documented
Submitted:	1 Oct 2008 13:26	Modified:	17 Jun 2010 19:42
Reporter:	Patrick Crews	Email Updates:
Status:	Closed	Impact on me:	None
Category:	Tools: MTR / mysql-test-run	Severity:	S3 (Non-critical)
Version:		OS:	Any
Assigned to:	Bjørn Munch	CPU Architecture:	Any
Tags:	Docs, mysql-test-run, remove_file, wildcard

Description:
mysql-test-run's remove_file command is fails when passed wildcards.  This behavior should be documented or altered.

While it could be argued that remove_file is behaving as documented - returns error 1 if the file does not exist, the fact that this is meant to replace the use of the OS's rm or del utilities implies that wildcards should be accepted.

How to repeat:
Place the following lines in a .test file and execute the following command:

./mtr --record <name_of_test_file>

#Test file content
--exec echo a > a.foo
--exec echo b > b.foo

--remove_file *.foo

You will see this error:
mysqltest: At line 4: command "remove_file" failed with error 1
 and the .foo files will still be present.

Suggested fix:
Either adjust the remove_file code to accept wildcards or make a note in documentation that wildcards (currently) do not work.

I feel that since this is mean to be a substitute for an OS's rm / del commands, the use of wildcards is implied.  We need to be explicit about the limitations of this function.

After consultation with Docs team, re-categorizing it to Server:Tests until we determine if this should be handled via changing mtr or just documenting the existing behavior

The remove_file in mysqltest has intentionally been written this way to make it very explicit _exactly_ which files are delete.

Please add to the documentation.

Recategorizing as Server: Documentation, assigning to myself.

Thank you for your bug report. This issue has been addressed in the documentation. The updated documentation will appear on our website shortly, and will be included in the next release of the relevant products.

Filename argument must evaluate to literal filename, not filename pattern.

Also true for chmod_file.

Magnus,

I am afraid that we will need the wildcard remove file thingy in order to fix sporadic failures in our tests. What is your rationale to require remove-file to list the file explicitly?

If you want to keep remove-file as it is, we would need a remove-file-wildcard that will take wildcards (but not recursive delete). If you are concerned about rm -rf / , we should make remove-file-wildcard operate only inside the vardir.

Please let me know.

Philip Stoev

I need this to be able to test error handling in partitioning (which can currently leave a lot of temporary files in a specific test case...) So I will implement remove_files which will take directory as first argument and optionally filenamepattern as a second argument. If the second argument is empty it will remove all files within that directory, otherwise it will only remove matching files. Just like list_files works, but instead of listing them, it will remove them.

Please notify Bjorn before coding.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/85855

3147 Mattias Jonsson	2009-10-06
      Bug#39774: mysql-test-run's remove_file can't use wildcards, this should be documented
      
      Needed to be able to clean up in an efficient way with wildcards.
      
      Added remove_files_wildcard that allows to remove multiple files
      at once.
      
      Removes only regular files (i.e. no directories and is not recursive).
      Works like list_files.
      
      Note: the pattern is match within the string, i.e. not matching start/end,
      so 'file*.txt' also matches list_file.txt for example. 
     @ client/mysqltest.cc
        Bug#39774: mysql-test-run's remove_file can't use wildcards, this should be documented
        
        Added remove_files_wildcard that allows to remove multiple files
        at once.
     @ mysql-test/r/mysqltest.result
        Bug#39774: mysql-test-run's remove_file can't use wildcards, this should be documented
        
        Updated test result
     @ mysql-test/t/mysqltest.test
        Bug#39774: mysql-test-run's remove_file can't use wildcards, this should be documented
        
        Added tests for remove_files_wildcard

This worked on Solaris but when I applied the patch in Windows and tried to run the mysqltest.test, I got

command "remove_files_wildcard" failed with error  -858993460

This is from the first test using file?.txt

Any idea?

BTW, that error code it reports is CCCCCCCC in hex

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/99049

2886 Bjorn Munch	2010-02-03
      Bug #39774 mysql-test-run's remove_file can't use wildcards, this should be documented
      Added remove_files_wildcard that allows to remove multiple files at once.
      This is a port of previous patch to Windows, plus resolving a merge 
      conflict with current 5.1-mtr branch.

Looks ok, awaiting patch that fixes Mattias comments.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/99223

2886 Bjorn Munch	2010-02-04
      Bug #39774 mysql-test-run's remove_file can't use wildcards, this should be documented
      Added remove_files_wildcard that allows to remove multiple files at once.
      This is a port of original patch to Windows.

Don't leave the non working code. Since we decided to change to "remove everything that is not a dir" it's no longer relevant since IS_DIR more or lees means !IS_REG

Checked the implementation of my_dir and yes, it does not set the IS_REG bit in the windows implementation.

I have filed Bug #50877 for the missing flag in my_dir().

As far as I can see, the only difference is that we won't skip "special files" like character or block devices or pipes, but I don't see this as a big problem.

I will follow Bug #50877 (since I'm the reporter) and can ask whoever picks it up to also undo my workaround.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/99266

2887 Bjorn Munch	2010-02-04
      Bug #39774 mysql-test-run's remove_file can't use wildcards, this should be documented
      Added remove_files_wildcard that allows to remove multiple files at once.
      This is a port of original patch to Windows.

Pushed to 5.1-mtr, trunk-mtr, next-mr-mtr, 6.0-codebase-mtr.

Sorry about the extra commit; silly bzr mistake

Pushed into 6.0.14-alpha (revid:alik@sun.com-20100220092844-jh07ayojyxn8uh9p) (version source revid:bjorn.munch@sun.com-20100207141526-5m83mmrz9ad2ouh1) (merge vers: 6.0.14-alpha) (pib:16)

Pushed into 5.5.3-m2 (revid:alik@sun.com-20100220092622-wvhh1vfy5tjq4mhu) (version source revid:bjorn.munch@sun.com-20100207113713-ng22k3dbavz6yohg) (merge vers: 5.5.2-m2) (pib:16)

Pushed into mysql-next-mr (revid:alik@sun.com-20100220092730-paoo5t9gcajs5dq8) (version source revid:bjorn.munch@sun.com-20100207114124-ath7uwrv9cjhpeb6) (pib:16)

No changelog entry needed.

Noted in 5.5.3, 6.0.14 changelogs.

mysqltest has a new remove_files_wildcard command that removes files
matching a pattern from a directory.  

Setting report to Need Merge pending push to 5.1.x, Celosia.

Pushed into 5.1.45 (revid:joro@sun.com-20100301083827-xnimmrjg6bh33o1o) (version source revid:azundris@mysql.com-20100222175719-viuh0f3gdsrkgv0r) (merge vers: 5.1.45) (pib:16)

Noted in 5.1.45 changelog.

A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/110637

3103 Martin Skold	2010-06-09 [merge]
      Merged in 5.1.45
      added:
        mysql-test/include/not_binlog_format_row.inc
        mysql-test/r/bug39022.result
        mysql-test/r/no_binlog.result
        mysql-test/r/partition_debug_sync.result
        mysql-test/std_data/bug48449.frm
        mysql-test/suite/rpl/r/rpl_slow_query_log.result
        mysql-test/suite/rpl/t/rpl_slow_query_log-slave.opt
        mysql-test/suite/rpl/t/rpl_slow_query_log.test
        mysql-test/t/bug39022.test
        mysql-test/t/no_binlog.test
        mysql-test/t/partition_debug_sync.test
      renamed:
        mysql-test/r/variables+c.result => mysql-test/r/variables_community.result
        mysql-test/t/variables+c.test => mysql-test/t/variables_community.test
      modified:
        client/mysql.cc
        client/mysql_upgrade.c
        client/mysqladmin.cc
        client/mysqlbinlog.cc
        client/mysqlcheck.c
        client/mysqldump.c
        client/mysqlimport.c
        client/mysqlshow.c
        client/mysqlslap.c
        client/mysqltest.cc
        cmd-line-utils/readline/rlmbutil.h
        cmd-line-utils/readline/text.c
        configure.in
        extra/yassl/include/yassl_error.hpp
        extra/yassl/src/ssl.cpp
        extra/yassl/src/yassl_error.cpp
        mysql-test/include/mtr_warnings.sql
        mysql-test/lib/My/ConfigFactory.pm
        mysql-test/lib/My/SafeProcess.pm
        mysql-test/lib/My/SafeProcess/safe_process_win.cc
        mysql-test/lib/mtr_cases.pm
        mysql-test/lib/mtr_gprof.pl
        mysql-test/lib/mtr_misc.pl
        mysql-test/lib/mtr_report.pm
        mysql-test/lib/mtr_stress.pl
        mysql-test/lib/v1/mtr_stress.pl
        mysql-test/lib/v1/mysql-test-run.pl
        mysql-test/mysql-test-run.pl
        mysql-test/r/archive.result
        mysql-test/r/backup.result
        mysql-test/r/bigint.result
        mysql-test/r/csv.result
        mysql-test/r/default.result
        mysql-test/r/delete.result
        mysql-test/r/fulltext.result
        mysql-test/r/func_gconcat.result
        mysql-test/r/func_time.result
        mysql-test/r/group_by.result
        mysql-test/r/group_min_max.result
        mysql-test/r/having.result
        mysql-test/r/innodb-autoinc.result
        mysql-test/r/innodb_mysql.result
        mysql-test/r/join.result
        mysql-test/r/log_state.result
        mysql-test/r/multi_update.result
        mysql-test/r/myisam.result
        mysql-test/r/mysqltest.result
        mysql-test/r/show_check.result
        mysql-test/r/sp-bugs.result
        mysql-test/r/sp-error.result
        mysql-test/r/sp.result
        mysql-test/r/sp_notembedded.result
        mysql-test/r/sp_trans.result
        mysql-test/r/subselect.result
        mysql-test/r/type_bit.result
        mysql-test/r/type_blob.result
        mysql-test/r/type_timestamp.result
        mysql-test/r/view.result
        mysql-test/r/view_grant.result
        mysql-test/r/warnings.result
        mysql-test/suite/rpl/r/rpl_sp.result
        mysql-test/suite/rpl/t/rpl_loaddata_symlink.test
        mysql-test/suite/sys_vars/r/log_basic.result
        mysql-test/suite/sys_vars/r/log_bin_trust_routine_creators_basic.result
        mysql-test/suite/sys_vars/r/slow_query_log_func.result
        mysql-test/suite/sys_vars/t/slow_query_log_func.test
        mysql-test/t/archive.test
        mysql-test/t/bigint.test
        mysql-test/t/csv.test
        mysql-test/t/delete.test
        mysql-test/t/fulltext.test
        mysql-test/t/func_gconcat.test
        mysql-test/t/group_by.test
        mysql-test/t/group_min_max.test
        mysql-test/t/having.test
        mysql-test/t/innodb-autoinc.test
        mysql-test/t/innodb_mysql.test
        mysql-test/t/join.test
        mysql-test/t/multi_update.test
        mysql-test/t/myisam.test
        mysql-test/t/mysqltest.test
        mysql-test/t/sp-bugs.test
        mysql-test/t/sp_notembedded.test
        mysql-test/t/subselect.test
        mysql-test/t/type_bit.test
        mysql-test/t/view.test
        mysql-test/t/view_grant.test
        mysys/default.c
        scripts/mysqld_multi.sh
        server-tools/instance-manager/options.cc
        sql/field.cc
        sql/ha_partition.cc
        sql/item.cc
        sql/item.h
        sql/item_cmpfunc.h
        sql/item_row.cc
        sql/item_sum.cc
        sql/item_timefunc.cc
        sql/log.cc
        sql/log_event.cc
        sql/log_event_old.cc
        sql/mysql_priv.h
        sql/mysqld.cc
        sql/opt_range.cc
        sql/opt_sum.cc
        sql/share/errmsg.txt
        sql/sp.cc
        sql/sp_cache.cc
        sql/sp_head.cc
        sql/sql_base.cc
        sql/sql_class.cc
        sql/sql_class.h
        sql/sql_lex.cc
        sql/sql_lex.h
        sql/sql_parse.cc
        sql/sql_repl.cc
        sql/sql_select.cc
        sql/sql_select.h
        sql/sql_table.cc
        sql/sql_trigger.cc
        sql/sql_update.cc
        sql/sql_view.cc
        sql/table.cc
        sql/table.h
        storage/archive/ha_archive.cc
        storage/csv/ha_tina.cc
        storage/innobase/buf/buf0buf.c
        storage/innobase/buf/buf0rea.c
        storage/innobase/handler/ha_innodb.cc
        storage/innobase/include/buf0rea.h
        storage/innobase/lock/lock0lock.c
        storage/innobase/os/os0file.c
        storage/myisam/ft_boolean_search.c
        storage/myisam/mi_dynrec.c
        storage/myisam/mi_locking.c
        support-files/mysql.spec.sh

Pushed into 5.1.47-ndb-7.0.16 (revid:martin.skold@mysql.com-20100617114014-bva0dy24yyd67697) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)

Pushed into 5.1.47-ndb-6.2.19 (revid:martin.skold@mysql.com-20100617115448-idrbic6gbki37h1c) (version source revid:martin.skold@mysql.com-20100609140708-52rvuyq4q500sxkq) (merge vers: 5.1.45-ndb-6.2.19) (pib:16)

Pushed into 5.1.47-ndb-6.3.35 (revid:martin.skold@mysql.com-20100617114611-61aqbb52j752y116) (version source revid:vasil.dimov@oracle.com-20100331130613-8ja7n0vh36a80457) (merge vers: 5.1.46) (pib:16)