Bug #41642 The purpose for the backup_engines suite isn't clear.
Submitted: 19 Dec 2008 15:23 Modified: 26 Mar 2009 23:44
Reporter: Chuck Bell Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Backup Severity:S3 (Non-critical)
Version:6.0.9 OS:Any
Assigned to: Chuck Bell CPU Architecture:Any

[19 Dec 2008 15:23] Chuck Bell
Description:
There is some confusion over what exactly is the intended use of the backup_engines suite.

Currently, the backup_engines suite is designed to select a different storage engine (myisam, memory, falcon, innodb) for a given test and cycle through the list. Thus, the suite can test backup interaction with tables created with these storage engines (one pass per engine). This is the limit of this suite.

However, using different storage engines in a given test is warranted for things like driver selection, mixing different drivers, and (in the case of MyISAM) exercising default versus native backup drivers (once permitted). But this suite does not solve those needs and should not be used for that or any similar purpose.

Therefore, the backup_engines suite is only usable for tests that test a single backup driver at a time -- which is to say a single storage engine at a time for creation of tables (currently, this is a 1:1 ratio but will not be so in the future). Thus the backup_engines suite has limited utility and should not be confused with testing all storage engines or even imply coverage of any multiple backup driver related testing.

Complicating this is the possibility that some may interpret the backup_engines suite as being a place to put tests that test all storage engines. This is false. The suite is not designed for that and given its single pass per storage engine and short list of storage engines is not suited for advanced testing of backup drivers and/or interaction with the storage engine layer (ha_* and below).

A greater problem comes when we will allow selection of drivers. The backup_engines suite is not designed for that possibility and should not be used in that capacity.  

How to repeat:
NA

Suggested fix:
Add copious comments to all of the backup suites which specify the exact nature and intended use of each suite. This can be in a README file but must be added to the backup documentation on the forge page (at the very least).

Rename the backup_engines suite to backup_select_engine. Examine the tests in the suite to determine their applicability given the limitations of the suite. If tests are found that exceed the limitations of the suite, modify them accordingly and/or move them to the appropriate suite.

Open a worklog to create a new suite named backup_drivers designed to allow selection of drivers (which will be needed once we allow user selected drivers). For example, tests could set table storage engine selection via setting variables (ENGINE1, ENGINE2, ENGINE3) and set backup driver selection via setting other variables (DRIVER1, DRIVER2, DRIVER3). In this way, a backup_drivers suite will do what some may think the backup_engines suite is doing now (but isn't) -- testing backup driver interaction with storage engines.
[2 Feb 2009 15:59] Chuck Bell
The backup_engines suite is designed to run a test once for each of the storage engines listed in the combinations file (stored in the root folder of the suite). The engines listed are falcon, myisam, innodb, and  memory. Thus, a given test runs once substituting the storage engine for the default-storage-engine setting. The tests written for this suite make assumptions about the default storage engine and the presumption that running the same test through each storage engine is in some way beneficial.

Problems
--------
* There are only 4 storage engines listed. All storage engines should be listed.
* Running the same test for each storage engine is inefficient and of little benefit. However, if we are testing backup for how it behaves for different storage engines, then a discrete test for each storage may (under very special cases) be beneficial.
* If you write a test that should be run against 'all' storage engines, then this suite will not meet your needs. A more elaborate test is needed to test the use of a single and multiple storage engines in the same backup or restore operation.

PROPOSAL
--------
The following is what I think should be done:

Option 1
--------
We agree the suite has some limited benefits. 

* Expand the combinations file to include all storage engines.
* Rename the suite to something else like backup_single_engine so that it reflects the true nature of the test suite.
* Examine the tests in the suite for compatibility with the existing limitations moving most if not all back to the backup suite.
* (optional and probably not worth the effort) Rework the suite to make it more efficient.

Option 2
--------
Eliminate the suite -- it is too limited and inefficient.

Option X
--------
Does anyone have another option?
[2 Feb 2009 16:06] Chuck Bell
Additional problem with the backup_engines suite:

* The --default-storage-engine parameter can be overridden and|or ignored.
[19 Feb 2009 14:59] Chuck Bell
The intital analysis is complete. There were some tests in the backup_engines suite that were not using the combinations file.

Initial pass at reorganization of tests (on Windows) show:

BEFORE: 105 tests executing in 622 seconds.
AFTER:   77 tests executing in 317 seconds.

Moved most tests out of backup_engines leaving only partition and trigger tests.
Created a new backup_ptr suite for testing point in time over replication modes rather than engines and replication modes (this was the greatest savings for number of tests).

Patch forthcoming. With all of the renames it is a challenge to create a patch that can be applied correctly.

Worked 20 hours. Estimate 8 hours max remain (including reviews).
[19 Feb 2009 19:20] Chuck Bell
There are now 74 tests down from 105. :)
[19 Feb 2009 19:21] Chuck Bell
Commands needed to apply patch

Attachment: 41642_linux.txt (text/plain), 1.95 KiB.

[19 Feb 2009 19:24] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/66957

2773 Chuck Bell	2009-02-19
      BUG#41642 : The purpose for the backup_engines suite isn't clear.
      
      This patch reorganizes the backup_engines tests. A new suite is created for those tests that must iterate over replication modes (backup_ptr). The storage engines list is extended to include all storage engines except for MERGE and FEDERATED which require special commands to utilize. 
      
      A description for each of the suites has been added to clarify the usage of the suites and what sort of tests should be included.
      
      Note: The archive storage engine is disabled for the partition tests. See BUG#43028 and BUG#43030 for details.
      removed:
        mysql-test/suite/backup_engines/r/backup_procedures.result
        mysql-test/suite/backup_engines/r/backup_ptr_commit_mixed.result
        mysql-test/suite/backup_engines/r/backup_ptr_commit_stmt.result
        mysql-test/suite/backup_engines/r/backup_ptr_mixed.result
        mysql-test/suite/backup_engines/r/backup_ptr_objects_mixed.result
        mysql-test/suite/backup_engines/r/backup_ptr_objects_row.result
        mysql-test/suite/backup_engines/r/backup_ptr_objects_stmt.result
        mysql-test/suite/backup_engines/r/backup_ptr_row.result
        mysql-test/suite/backup_engines/r/backup_ptr_stmt.result
        mysql-test/suite/backup_engines/t/backup_procedures.test
        mysql-test/suite/backup_engines/t/backup_ptr_commit_mixed.test
        mysql-test/suite/backup_engines/t/backup_ptr_commit_row.test
        mysql-test/suite/backup_engines/t/backup_ptr_commit_stmt.test
        mysql-test/suite/backup_engines/t/backup_ptr_mixed.test
        mysql-test/suite/backup_engines/t/backup_ptr_objects_mixed.test
        mysql-test/suite/backup_engines/t/backup_ptr_objects_row.test
        mysql-test/suite/backup_engines/t/backup_ptr_objects_stmt.test
        mysql-test/suite/backup_engines/t/backup_ptr_row.test
        mysql-test/suite/backup_engines/t/backup_ptr_stmt.test
        mysql-test/suite/backup_engines/t/disabled.def
      added:
        mysql-test/suite/backup/README
        mysql-test/suite/backup_engines/README
        mysql-test/suite/backup_engines/include/not_archive.inc
        mysql-test/suite/backup_engines/include/not_csv.inc
        mysql-test/suite/backup_ptr/
        mysql-test/suite/backup_ptr/README
        mysql-test/suite/backup_ptr/include/
        mysql-test/suite/backup_ptr/r/
        mysql-test/suite/backup_ptr/r/backup_ptr.result
        mysql-test/suite/backup_ptr/r/backup_ptr_objects.result
        mysql-test/suite/backup_ptr/t/
      renamed:
        mysql-test/suite/backup_engines/include/backup_ptr.inc => mysql-test/suite/backup_ptr/t/backup_ptr.test
        mysql-test/suite/backup_engines/include/backup_ptr_commit.inc => mysql-test/suite/backup_ptr/t/backup_ptr_commits.test
        mysql-test/suite/backup_engines/include/backup_ptr_objects.inc => mysql-test/suite/backup_ptr/t/backup_ptr_objects.test
        mysql-test/suite/backup_engines/include/not_have_falcon.inc => mysql-test/suite/backup_engines/include/not_falcon.inc
        mysql-test/suite/backup_engines/include/not_have_innodb.inc => mysql-test/suite/backup_engines/include/not_innodb.inc
        mysql-test/suite/backup_engines/include/not_have_memory.inc => mysql-test/suite/backup_engines/include/not_memory.inc
        mysql-test/suite/backup_engines/include/not_have_myisam.inc => mysql-test/suite/backup_engines/include/not_myisam.inc
        mysql-test/suite/backup_engines/r/backup_functions.result => mysql-test/suite/backup/r/backup_functions.result
        mysql-test/suite/backup_engines/r/backup_online_testing.result => mysql-test/suite/backup/r/backup_online_testing.result
        mysql-test/suite/backup_engines/r/backup_ptr_commit_row.result => mysql-test/suite/backup_ptr/r/backup_ptr_commits.result
        mysql-test/suite/backup_engines/r/backup_tmp_tables.result => mysql-test/suite/backup/r/backup_tmp_tables.result
        mysql-test/suite/backup_engines/t/backup_functions.test => mysql-test/suite/backup/t/backup_functions.test
        mysql-test/suite/backup_engines/t/backup_online_testing.test => mysql-test/suite/backup/t/backup_online_testing.test
        mysql-test/suite/backup_engines/t/backup_tmp_tables.test => mysql-test/suite/backup/t/backup_tmp_tables.test
      modified:
        mysql-test/suite/backup/r/backup_procedures.result
        mysql-test/suite/backup/t/backup_procedures.test
        mysql-test/suite/backup_engines/combinations
        mysql-test/suite/backup_engines/r/backup_partition.result
        mysql-test/suite/backup_engines/t/backup_partition.test
        mysql-test/suite/backup_engines/t/backup_partitioning.test
        mysql-test/suite/backup_engines/t/backup_triggers.test
        mysql-test/suite/backup_ptr/t/backup_ptr.test
        mysql-test/suite/backup_ptr/t/backup_ptr_commits.test
        mysql-test/suite/backup_ptr/t/backup_ptr_objects.test
        mysql-test/suite/backup_engines/include/not_falcon.inc
        mysql-test/suite/backup_engines/include/not_innodb.inc
        mysql-test/suite/backup_engines/include/not_memory.inc
        mysql-test/suite/backup_engines/include/not_myisam.inc
        mysql-test/suite/backup/r/backup_functions.result
        mysql-test/suite/backup_ptr/r/backup_ptr_commits.result
        mysql-test/suite/backup/t/backup_functions.test
[23 Feb 2009 13:59] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/67194

2773 Chuck Bell	2009-02-23
      BUG#41642 : The purpose for the backup_engines suite isn't clear.
      
      This patch reorganizes the backup_engines tests. A new suite is 
      created for those tests that must iterate over replication modes 
      (backup_ptr). The storage engines list is extended to include all 
      storage engines except for MERGE and FEDERATED which require 
      special commands to utilize. 
      
      A description for each of the suites has been added to clarify 
      the usage of the suites and what sort of tests should be included.
      removed:
        mysql-test/suite/backup_engines/r/backup_procedures.result
        mysql-test/suite/backup_engines/r/backup_ptr_commit_mixed.result
        mysql-test/suite/backup_engines/r/backup_ptr_commit_stmt.result
        mysql-test/suite/backup_engines/r/backup_ptr_mixed.result
        mysql-test/suite/backup_engines/r/backup_ptr_objects_mixed.result
        mysql-test/suite/backup_engines/r/backup_ptr_objects_row.result
        mysql-test/suite/backup_engines/r/backup_ptr_objects_stmt.result
        mysql-test/suite/backup_engines/r/backup_ptr_row.result
        mysql-test/suite/backup_engines/r/backup_ptr_stmt.result
        mysql-test/suite/backup_engines/t/backup_procedures.test
        mysql-test/suite/backup_engines/t/backup_ptr_commit_mixed.test
        mysql-test/suite/backup_engines/t/backup_ptr_commit_row.test
        mysql-test/suite/backup_engines/t/backup_ptr_commit_stmt.test
        mysql-test/suite/backup_engines/t/backup_ptr_mixed.test
        mysql-test/suite/backup_engines/t/backup_ptr_objects_mixed.test
        mysql-test/suite/backup_engines/t/backup_ptr_objects_row.test
        mysql-test/suite/backup_engines/t/backup_ptr_objects_stmt.test
        mysql-test/suite/backup_engines/t/backup_ptr_row.test
        mysql-test/suite/backup_engines/t/backup_ptr_stmt.test
        mysql-test/suite/backup_engines/t/disabled.def
      added:
        mysql-test/suite/backup/README
        mysql-test/suite/backup_engines/README
        mysql-test/suite/backup_engines/include/not_archive.inc
        mysql-test/suite/backup_engines/include/not_csv.inc
        mysql-test/suite/backup_ptr/
        mysql-test/suite/backup_ptr/README
        mysql-test/suite/backup_ptr/combinations
        mysql-test/suite/backup_ptr/include/
        mysql-test/suite/backup_ptr/r/
        mysql-test/suite/backup_ptr/r/backup_ptr.result
        mysql-test/suite/backup_ptr/r/backup_ptr_objects.result
        mysql-test/suite/backup_ptr/t/
      renamed:
        mysql-test/suite/backup_engines/include/backup_ptr.inc => mysql-test/suite/backup_ptr/t/backup_ptr.test
        mysql-test/suite/backup_engines/include/backup_ptr_commit.inc => mysql-test/suite/backup_ptr/t/backup_ptr_commits.test
        mysql-test/suite/backup_engines/include/backup_ptr_objects.inc => mysql-test/suite/backup_ptr/t/backup_ptr_objects.test
        mysql-test/suite/backup_engines/include/not_have_falcon.inc => mysql-test/suite/backup_engines/include/not_falcon.inc
        mysql-test/suite/backup_engines/include/not_have_innodb.inc => mysql-test/suite/backup_engines/include/not_innodb.inc
        mysql-test/suite/backup_engines/include/not_have_memory.inc => mysql-test/suite/backup_engines/include/not_memory.inc
        mysql-test/suite/backup_engines/include/not_have_myisam.inc => mysql-test/suite/backup_engines/include/not_myisam.inc
        mysql-test/suite/backup_engines/r/backup_functions.result => mysql-test/suite/backup/r/backup_functions.result
        mysql-test/suite/backup_engines/r/backup_online_testing.result => mysql-test/suite/backup/r/backup_online_testing.result
        mysql-test/suite/backup_engines/r/backup_ptr_commit_row.result => mysql-test/suite/backup_ptr/r/backup_ptr_commits.result
        mysql-test/suite/backup_engines/r/backup_tmp_tables.result => mysql-test/suite/backup/r/backup_tmp_tables.result
        mysql-test/suite/backup_engines/t/backup_functions.test => mysql-test/suite/backup/t/backup_functions.test
        mysql-test/suite/backup_engines/t/backup_online_testing.test => mysql-test/suite/backup/t/backup_online_testing.test
        mysql-test/suite/backup_engines/t/backup_tmp_tables.test => mysql-test/suite/backup/t/backup_tmp_tables.test
      modified:
        mysql-test/suite/backup/r/backup_procedures.result
        mysql-test/suite/backup/t/backup_procedures.test
        mysql-test/suite/backup_engines/combinations
        mysql-test/suite/backup_engines/r/backup_partition.result
        mysql-test/suite/backup_engines/t/backup_partition.test
        mysql-test/suite/backup_engines/t/backup_partitioning.test
        mysql-test/suite/backup_engines/t/backup_triggers.test
        mysql-test/suite/backup_ptr/t/backup_ptr.test
        mysql-test/suite/backup_ptr/t/backup_ptr_commits.test
        mysql-test/suite/backup_ptr/t/backup_ptr_objects.test
        mysql-test/suite/backup_engines/include/not_falcon.inc
        mysql-test/suite/backup_engines/include/not_innodb.inc
        mysql-test/suite/backup_engines/include/not_memory.inc
        mysql-test/suite/backup_engines/include/not_myisam.inc
        mysql-test/suite/backup/r/backup_functions.result
        mysql-test/suite/backup_ptr/r/backup_ptr_commits.result
        mysql-test/suite/backup/t/backup_functions.test
[5 Mar 2009 6:50] Rafal Somla
Patch approved.
[5 Mar 2009 20:05] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68417

2784 Chuck Bell	2009-03-05
      BUG#41642 : The purpose for the backup_engines suite isn't clear.
      
      This patch reorganizes the backup_engines tests. A new suite is created
      for those tests that must iterate over replication modes (backup_ptr). 
      The storage engines list is extended to include all storage engines 
      except for MERGE and FEDERATED which require special commands to utilize. 
      
      A description for each of the suites has been added to clarify the usage 
      of the suites and what sort of tests should be included.
      
      Note: The archive storage engine is disabled for the partition tests. 
      See BUG#43028 and BUG#43030 for details.
      
      The following are commands needed to apply this patch. These create the 
      new folders and move the test files. The changes to the test files are 
      included in the patch. These should be executed from the root of the 
      source tree.
      
      Added backup_ptr to the default list in new mtr.
      removed:
        mysql-test/suite/backup_engines/r/backup_procedures.result
        mysql-test/suite/backup_engines/r/backup_ptr_commit_mixed.result
        mysql-test/suite/backup_engines/r/backup_ptr_commit_stmt.result
        mysql-test/suite/backup_engines/r/backup_ptr_mixed.result
        mysql-test/suite/backup_engines/r/backup_ptr_objects_mixed.result
        mysql-test/suite/backup_engines/r/backup_ptr_objects_row.result
        mysql-test/suite/backup_engines/r/backup_ptr_objects_stmt.result
        mysql-test/suite/backup_engines/r/backup_ptr_row.result
        mysql-test/suite/backup_engines/r/backup_ptr_stmt.result
        mysql-test/suite/backup_engines/t/backup_procedures.test
        mysql-test/suite/backup_engines/t/backup_ptr_commit_mixed.test
        mysql-test/suite/backup_engines/t/backup_ptr_commit_row.test
        mysql-test/suite/backup_engines/t/backup_ptr_commit_stmt.test
        mysql-test/suite/backup_engines/t/backup_ptr_mixed.test
        mysql-test/suite/backup_engines/t/backup_ptr_objects_mixed.test
        mysql-test/suite/backup_engines/t/backup_ptr_objects_row.test
        mysql-test/suite/backup_engines/t/backup_ptr_objects_stmt.test
        mysql-test/suite/backup_engines/t/backup_ptr_row.test
        mysql-test/suite/backup_engines/t/backup_ptr_stmt.test
        mysql-test/suite/backup_engines/t/disabled.def
      added:
        mysql-test/suite/backup_ptr/
        mysql-test/suite/backup_ptr/include/
        mysql-test/suite/backup_ptr/r/
        mysql-test/suite/backup_ptr/t/
      renamed:
        mysql-test/suite/backup_engines/include/backup_ptr.inc => mysql-test/suite/backup_ptr/t/backup_ptr.test
        mysql-test/suite/backup_engines/include/backup_ptr_commit.inc => mysql-test/suite/backup_ptr/t/backup_ptr_commits.test
        mysql-test/suite/backup_engines/include/backup_ptr_objects.inc => mysql-test/suite/backup_ptr/t/backup_ptr_objects.test
        mysql-test/suite/backup_engines/include/not_have_falcon.inc => mysql-test/suite/backup_engines/include/not_falcon.inc
        mysql-test/suite/backup_engines/include/not_have_innodb.inc => mysql-test/suite/backup_engines/include/not_innodb.inc
        mysql-test/suite/backup_engines/include/not_have_memory.inc => mysql-test/suite/backup_engines/include/not_memory.inc
        mysql-test/suite/backup_engines/include/not_have_myisam.inc => mysql-test/suite/backup_engines/include/not_myisam.inc
        mysql-test/suite/backup_engines/r/backup_functions.result => mysql-test/suite/backup/r/backup_functions.result
        mysql-test/suite/backup_engines/r/backup_online_testing.result => mysql-test/suite/backup/r/backup_online_testing.result
        mysql-test/suite/backup_engines/r/backup_ptr_commit_row.result => mysql-test/suite/backup_ptr/r/backup_ptr_commits.result
        mysql-test/suite/backup_engines/r/backup_tmp_tables.result => mysql-test/suite/backup/r/backup_tmp_tables.result
        mysql-test/suite/backup_engines/t/backup_functions.test => mysql-test/suite/backup/t/backup_functions.test
        mysql-test/suite/backup_engines/t/backup_online_testing.test => mysql-test/suite/backup/t/backup_online_testing.test
        mysql-test/suite/backup_engines/t/backup_tmp_tables.test => mysql-test/suite/backup/t/backup_tmp_tables.test
      modified:
        mysql-test/mysql-test-run.pl
        mysql-test/suite/backup/r/backup_procedures.result
        mysql-test/suite/backup/t/backup_procedures.test
        mysql-test/suite/backup_engines/combinations
        mysql-test/suite/backup_engines/r/backup_partition.result
        mysql-test/suite/backup_engines/t/backup_partition.test
        mysql-test/suite/backup_engines/t/backup_partitioning.test
        mysql-test/suite/backup_engines/t/backup_triggers.test
        mysql-test/suite/backup_ptr/t/backup_ptr.test
        mysql-test/suite/backup_ptr/t/backup_ptr_commits.test
        mysql-test/suite/backup_ptr/t/backup_ptr_objects.test
        mysql-test/suite/backup_engines/include/not_falcon.inc
        mysql-test/suite/backup_engines/include/not_innodb.inc
        mysql-test/suite/backup_engines/include/not_memory.inc
        mysql-test/suite/backup_engines/include/not_myisam.inc
        mysql-test/suite/backup/r/backup_functions.result
        mysql-test/suite/backup_ptr/r/backup_ptr_commits.result
        mysql-test/suite/backup/t/backup_functions.test
[5 Mar 2009 20:41] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68420

2785 Chuck Bell	2009-03-05
      BUG#41642 : The purpose for the backup_engines suite isn't clear.
      
      Added missing includes.
      added:
        mysql-test/suite/backup_engines/include/not_archive.inc
        mysql-test/suite/backup_engines/include/not_csv.inc
[5 Mar 2009 20:51] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68422

2786 Chuck Bell	2009-03-05
      BUG#41642 : The purpose for the backup_engines suite isn't clear.
      
      Added missing files from patch. 
      added:
        mysql-test/suite/backup/README
        mysql-test/suite/backup_engines/README
        mysql-test/suite/backup_ptr/README
        mysql-test/suite/backup_ptr/combinations
        mysql-test/suite/backup_ptr/r/backup_ptr.result
[5 Mar 2009 20:57] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68424

2787 Chuck Bell	2009-03-05
      BUG#41642 : The purpose for the backup_engines suite isn't clear.
      
      Ok, I think I found them all now.
      added:
        mysql-test/suite/backup_ptr/r/backup_ptr_objects.result
[26 Mar 2009 12:33] Bugs System
Pushed into 6.0.11-alpha (revid:alik@sun.com-20090326121822-pt84kzxxayzho4mn) (version source revid:charles.bell@sun.com-20090309134019-vr480npdp30f2gvh) (merge vers: 6.0.11-alpha) (pib:6)
[26 Mar 2009 23:44] Paul DuBois
Test suite changes. No changelog entry needed.