Bug #17823 'arc' directories inside database directories
Submitted: 1 Mar 2006 13:08 Modified: 17 Oct 2008 17:08
Reporter: Beat Vontobel (Silver Quality Contributor) (OCA) Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Views Severity:S3 (Non-critical)
Version:5.0.18 OS:Any (any)
Assigned to: Gleb Shchepa
Tags: bfsm_2007_10_18, qc
Triage: Triaged: D4 (Minor)

[1 Mar 2006 13:08] Beat Vontobel
Description:
I don't know if this is just a documentation issue or maybe a bug (temporary files not deleted): Since some time I noticed some subdirectories named 'arc' popping up in my database directories. It seems as they only contain old versions of view definitions (backup of .frm files) numbered sequentially (0001, 0002...). Some of those files are pretty old (dating back until last december). 

Questions arise: What's the exact purpose of these files? Just a backup of older versions of views? Is this behaviour configurable? Or are these just left over temporary files that haven't been deleted by the server (and thus a bug)? Can these files be safely deleted manually? Or should they even be backed up together with the table files when for example doing a mysqlhotcopy?

I checked the chapter on views in the most recent version of the manual, did a search for 'arc' on the manual and on the bugs system but didn't find anything. So this should at least be documented in an appropriate place.

How to repeat:
-

Suggested fix:
Document behaviour/purpose and/or make the server delete these files if they are no longer needed.
[1 Mar 2006 13:43] Valerii Kravchuk
Thank you for a problem report. Can you provide a repeatable sequence of actions that leads to creating of that "arc" subdirectory?
[1 Mar 2006 14:05] Beat Vontobel
Hi Valeriy, unfortunately not yet. I don't have binary logs dating that far back. But there are some of the files that have dates when 5.0.18 was already running (most of them however are still from 5.0.17 or earlier).

The files were created by the user which mysqld is running with and nobody has access to the data directory. All the files correspond to views we have (or once had) on the database. I got similar files on more than one server. Master and slaves are both affected. I post a content list of those arc directories from our master server below.

Can I guess from your reaction that this is not merely a documentation issue? The files shouldn't be there?

I'd just need a hint from you or any other developer on when mysqld creates one of these files in the first place. Then I'm sure I can make a repeatable test case to show when these (possibly temporary files) are not deleted again. Right now I just don't have no idea what these files are all about or if they even _should_ be there. This makes it difficult to come up with a test case.

Thanks,
Beat

mysql1:/var/lib/mysql # find . -name arc -exec ls -lah \{\} \;
total 13K
drwx------    2 mysql    daemon        192 Jan  5 16:37 .
drwxr-x---    3 mysql    daemon       1.2K Feb 28 15:40 ..
-rw-rw----    1 mysql    daemon       2.4K Dec 21 17:56 swissmetnet_sdm_1h.frm-0002
-rw-rw----    1 mysql    daemon       2.9K Jan  5 16:33 swissmetnet_sdm_1h.frm-0003
-rw-rw----    1 mysql    daemon       2.9K Jan  5 16:35 swissmetnet_sdm_1h.frm-0004
total 29K
drwx------    2 mysql    daemon        304 Nov  1 00:17 .
drwxr-x---    3 mysql    daemon        736 Feb 27 13:33 ..
-rw-rw----    1 mysql    daemon       1.3K Oct 31 15:54 laender.frm-0002
-rw-rw----    1 mysql    daemon       1.3K Oct 31 16:10 laender.frm-0003
-rw-rw----    1 mysql    daemon       1.3K Oct 31 23:25 laender.frm-0004
-rw-rw----    1 mysql    daemon       2.7K Oct 31 23:25 stationen.frm-0005
-rw-rw----    1 mysql    daemon       2.8K Oct 31 23:52 stationen.frm-0006
-rw-rw----    1 mysql    daemon       2.8K Nov  1 00:05 stationen.frm-0007
-rw-rw----    1 mysql    daemon       2.7K Oct 31 15:19 stationen_view.frm-0001
total 5.5K
drwx------    2 mysql    daemon         96 Feb 27 13:05 .
drwx------    3 mysql    daemon       1.9K Feb 27 13:05 ..
-rw-rw----    1 mysql    daemon       1.3K Feb 27 11:31 stations_grouped.frm-0001
total 21K
drwx------    2 mysql    daemon        288 Feb 17 01:32 .
drwx------    3 mysql    daemon       1.1K Feb 28 15:33 ..
-rw-rw----    1 mysql    daemon       1.2K Feb 17 00:10 climate_stations.frm-0001
-rw-rw----    1 mysql    daemon       1.2K Feb 17 01:28 climate_stations.frm-0002
-rw-rw----    1 mysql    daemon       1.4K Dec 20 15:24 swissmetnet_holes.frm-0001
-rw-rw----    1 mysql    daemon       1.3K Dec 20 15:24 swissmetnet_holes.frm-0002
-rw-rw----    1 mysql    daemon       1.5K Dec 20 15:25 swissmetnet_holes.frm-0003
total 94K
drwx------    2 mysql    daemon        920 Jan  5 16:38 .
drwxr-x---    3 mysql    daemon       5.1K Feb 28 15:42 ..
-rw-rw----    1 mysql    daemon       1.4K Nov 30 15:26 anetz10_test.frm-0004
-rw-rw----    1 mysql    daemon       1.6K Nov 30 15:28 anetz10_test.frm-0005
-rw-rw----    1 mysql    daemon       2.1K Nov 30 15:29 anetz10_test.frm-0006
-rw-rw----    1 mysql    daemon       1.8K Nov 22 22:22 astro_daylight.frm-0001
-rw-rw----    1 mysql    daemon       1.8K Nov 22 22:23 astro_daylight.frm-0002
-rw-rw----    1 mysql    daemon       1.8K Nov 22 22:24 astro_daylight.frm-0003
-rw-rw----    1 mysql    daemon       1.8K Nov  9 20:20 astro_daylight.frm-0028
-rw-rw----    1 mysql    daemon       1.8K Nov  9 20:25 astro_daylight.frm-0029
-rw-rw----    1 mysql    daemon       1.8K Nov  9 20:27 astro_daylight.frm-0030
-rw-rw----    1 mysql    daemon       1.4K Nov  9 17:47 astro_daytime.frm-0002
-rw-rw----    1 mysql    daemon       1.2K Nov  9 17:48 astro_daytime.frm-0003
-rw-rw----    1 mysql    daemon       1.4K Nov  9 18:07 astro_daytime.frm-0004
-rw-rw----    1 mysql    daemon       4.0K Nov  9 19:22 astro_local.frm-0008
-rw-rw----    1 mysql    daemon       4.0K Nov  9 19:22 astro_local.frm-0009
-rw-rw----    1 mysql    daemon       3.0K Nov  9 19:23 astro_local.frm-0010
-rw-rw----    1 mysql    daemon       2.7K Dec 22 22:49 swissmetnet_sdm_1h.frm-0004
-rw-rw----    1 mysql    daemon       2.7K Dec 22 23:04 swissmetnet_sdm_1h.frm-0005
-rw-rw----    1 mysql    daemon       2.8K Dec 22 23:35 swissmetnet_sdm_1h.frm-0006
-rw-rw----    1 mysql    daemon       1.6K Dec 22 23:16 test.frm-0002
-rw-rw----    1 mysql    daemon       1.7K Dec 22 23:17 test.frm-0003
-rw-rw----    1 mysql    daemon       1.6K Dec 22 23:18 test.frm-0004
-rw-rw----    1 mysql    daemon        444 Nov 22 22:36 v.frm-0001
[1 Mar 2006 15:07] Beat Vontobel
I just checked the source (5.0.18): Some functions in parse_file.cc manipulating .frm files take an argument on how many back copies of .frm files for views to keep. Then they check for the existence of an arc directory, copy the backup and delete older versions. Also in sql_db.cc there is a function mysql_rm_arc_files() to remove those archived .frm file (for example on deletion of a database).

So I really think this is intended behaviour and probably a documentation only issue: Are the files just for my convenience? Or does the server need them? Maybe I'm looking in all the wrong places but I just couldn't find anything in the manual. (I even did a SHOW VARIABLES LIKE '%arc%' to see if there maybe is a setting to control this.)
[5 Mar 2006 18:00] Markus Popp
I also have some arc directories with .frm-xxxx (where xxxx are continuous numbers) in it. The .frm files are identical in structure with .frm files for views in the directory one level higher. Since the files inside the arc directory are all older than the corresponding files in the directory above, I suppose that they are kind of backup files.

However, I agree that this behaviour should be documented.
[6 Mar 2006 7:36] Valerii Kravchuk
The purpose of this 'arc' subdirectories should be clearly explained in the manual.
[15 Nov 2007 18:47] Timothy Smith
Closed, see bug #4821.
[15 Nov 2007 20:55] Timothy Smith
Sorry, re-opening this to document the existence of the .frm archive folder which is added during DDL on a VIEW.
[10 Jan 2008 18:16] MC Brown
The documentation has been updated to note the creation and existence of these temporary files when creating new views.
[27 Mar 2008 6:35] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/44498

ChangeSet@1.2604, 2008-03-27 10:33:13+04:00, gshchepa@host.loc +2 -0
  Fixed bug #17823: 'arc' directories inside database directories.
  
  Server creates "arc" directories inside database directories and
  maintains there useless copies of .frm files.
  
  Creation and renaming procedures of those copies as well as
  creation of "arc" directories has been discontinued.
  Removal procedure has been kept untouched to be able to
  cleanup existent database directories by the DROP DATABASE
  query.
[27 Mar 2008 8:27] Beat Vontobel
Thanks for the thorough documentation of this feature -- it was after all really a minor issue, but I get annoyed by anything that happens in my database directories that is not fully understood. Here it was just the documentation that made the difference between a bug and a feature :)
[24 Sep 2008 18:43] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/54496

2677 Gleb Shchepa	2008-09-24
      Fixed bug #17823: 'arc' directories inside database directories.
      
      Server created "arc" directories inside database directories and
      maintained there useless copies of .frm files.
      
      Creation and renaming procedures of those copies as well as
      creation of "arc" directories has been discontinued.
      Removal procedure has been kept untouched to be able to
      cleanup existent database directories by the DROP DATABASE
      query. Also view renaming procedure has been updated to remove
      these directories.
[30 Sep 2008 14:05] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/54781

2686 Gleb Shchepa	2008-09-30
      Fixed bug #17823: 'arc' directories inside database directories.
      
      Server created "arc" directories inside database directories and
      maintained there useless copies of .frm files.
      
      Creation and renaming procedures of those copies as well as
      creation of "arc" directories has been discontinued.
      Removal procedure has been kept untouched to be able to
      cleanup existent database directories by the DROP DATABASE
      query. Also view renaming procedure has been updated to remove
      these directories.
[30 Sep 2008 14:05] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/54782

2686 Gleb Shchepa	2008-09-30
      Fixed bug #17823: 'arc' directories inside database directories.
      
      Server created "arc" directories inside database directories and
      maintained there useless copies of .frm files.
      
      Creation and renaming procedures of those copies as well as
      creation of "arc" directories has been discontinued.
      Removal procedure has been kept untouched to be able to
      cleanup existent database directories by the DROP DATABASE
      query. Also view renaming procedure has been updated to remove
      these directories.
[9 Oct 2008 11:46] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/55916

2705 Rafal Somla	2008-10-09
      Disabling tests affected by BUG#17823.
[9 Oct 2008 18:07] Bugs System
Pushed into 5.1.30  (revid:gshchepa@mysql.com-20080930125028-7gias3z5rxc8tgnw) (version source revid:gshchepa@mysql.com-20080930131611-jqto3jk100xv7bj8) (pib:4)
[15 Oct 2008 14:55] Paul Dubois
This is actually pushed to 5.1.29, not 5.1.30.
[15 Oct 2008 16:15] Paul Dubois
Noted in 5.1.29 changelog.

In connection with view creation, the server created arc directories
inside database directories and maintained useless copies of .frm
files there. Creation and renaming procedures of those copies as well
as creation of arc directories has been discontinued.

This change does cause a problem when downgrading to older server
versions which manifests itself under these circumstances:

1) Create a view v_orig in MySQL 5.1.29 or higher.

2) Rename the view to v_new and then back to v_orig.

3) Downgrade to an older 5.1.X server and run mysql_upgrade.

4) Try to rename v_orig to v_new again. This operation fails.

As a workaround to avoid this problem, use either of these
approaches: 

* Dump your data using mysqldump before downgrading and reload the dump
file after downgrading.

* Instead of renaming a view after the downgrade, drop it and recreate
it.     

Setting report to NDI pending push into 6.0.x.
[17 Oct 2008 16:42] Bugs System
Pushed into 6.0.8-alpha  (revid:gshchepa@mysql.com-20080930125028-7gias3z5rxc8tgnw) (version source revid:gshchepa@mysql.com-20080930133141-9dgehzh4oy65rbjk) (pib:5)
[17 Oct 2008 17:08] Paul Dubois
Noted in 6.0.8 changelog.
[17 Oct 2008 17:24] Bugs System
Pushed into 6.0.8-alpha  (revid:rafal.somla@sun.com-20081009114540-xafueo9wlo92dpmm) (version source revid:rafal.somla@sun.com-20081016091048-sjyw13rbvolvk2p8) (pib:5)
[28 Oct 2008 21:02] Bugs System
Pushed into 5.1.29-ndb-6.2.17  (revid:gshchepa@mysql.com-20080930125028-7gias3z5rxc8tgnw) (version source revid:tomas.ulin@sun.com-20081028140209-u4emkk1xphi5tkfb) (pib:5)
[28 Oct 2008 22:21] Bugs System
Pushed into 5.1.29-ndb-6.3.19  (revid:gshchepa@mysql.com-20080930125028-7gias3z5rxc8tgnw) (version source revid:tomas.ulin@sun.com-20081028194045-0353yg8cvd2c7dd1) (pib:5)
[1 Nov 2008 9:46] Bugs System
Pushed into 5.1.29-ndb-6.4.0  (revid:gshchepa@mysql.com-20080930125028-7gias3z5rxc8tgnw) (version source revid:jonas@mysql.com-20081101082305-qx5a1bj0z7i8ueys) (pib:5)
[20 Nov 2008 14:05] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/59382

2742 Jorgen Loland	2008-11-20
      Bug#17823 - 'arc' directories inside database directories
      
      Reenable backup_no_data.test that failed due to this bug.
[20 Nov 2008 14:10] Rafal Somla
OK to re-enable backup_no_data test now.
[21 Nov 2008 8:29] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/59473

2743 Jorgen Loland	2008-11-21
      Bug#17823 - 'arc' directories inside database directories
      
      Reenabled tests still fail. Revert of previous patch.
[26 Nov 2008 8:50] Bugs System
Pushed into 6.0.9-alpha  (revid:jorgen.loland@sun.com-20081121083124-22nb4c7oikn0h44n) (version source revid:jorgen.loland@sun.com-20081126084449-lq9ckif3hmxesy1n) (pib:5)
[4 Dec 2008 20:20] Paul Dubois
Noted in 5.0.72 changelog.
[4 Dec 2008 20:21] Paul Dubois
The downgrade problem introduced by the fix for this bug is addressed as Bug#40021.
[10 Dec 2008 5:41] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/61141

2742 He Zhenxing	2008-12-10 [merge]
      Auto Merge