Bug #47698 backup.backup_logs fails under valgrind
Submitted: 29 Sep 2009 6:59 Modified: 20 Feb 2010 19:07
Reporter: Alexander Nozdrin Email Updates:
Status: Closed Impact on me:
None 
Category:Tests: Backup Severity:S3 (Non-critical)
Version:6.0 OS:Any
Assigned to: Chuck Bell CPU Architecture:Any
Tags: experimental, pb2, sporadic, test failure

[29 Sep 2009 6:59] Alexander Nozdrin
Description:
This bug is different from Bug#46952.

Symptoms:

@@ -210,7 +210,7 @@
 are same.
 SELECT timediff(@stop_backup, @start_backup) > 5;
 timediff(@stop_backup, @start_backup) > 5
-0
+1

How to repeat:
http://pb2.norway.sun.com/web.py?template=mysql_show_test_failure&test_failure_id=2348626&...
[29 Sep 2009 6:59] Alexander Nozdrin
The test case has been marked experimental.
[10 Nov 2009 15:23] Chuck Bell
The test contains an arbitrary time calculation that has no basis. In this case, the test is comparing the start and stop times by a factor of '5'.

We must either remove the arbitrary calculation or introduce a means to delay backup so that we can guarantee time will elapse. To do so, one must use a time delay mechanism like sleep or real_sleep while backup is being run. However, it is know that these mechanisms are not reliable and therefore one cannot predict exactly how long a process will 'sleep' and therefore cannot be made predictable.

The arbitrary calculation is line 296:
 
--echo We calculate the timedifference between backup start time and stop
--echo time. If this difference is '0', then backup start time and stop time
--echo are same.

SELECT timediff(@stop_backup, @start_backup) > 5;

The benefit of doing this check is dubious. This test is stating that the stop time of a backup operation is at least 5 seconds more than the start time. But that is not reliable. If run on a fast system then it is possible a simple backup can run in less than 5 seconds. If this happens, the result will be 0 (as recorded in the result file). But if run on a very slow machine (like valgrind on PB2) it could be much more than 5 seconds and the result will be 1. But the real problem is the test statement verifies nothing and has no meaning for quality given the limited predictability of execution on slower machines. 

Furthermore, the comment that goes with the test statement is wrong. The test statement is not checking for start and stop to be = 0.

Based on this, the following is the solution for this bug.

Solution
--------
Remove the arbitrary time calculation from the test. It is not needed and serves no purpose.

Note: Only 1 reviewer is needed since this patch will change on test-related files.
[10 Nov 2009 15:25] Chuck Bell
There are more arbitrary time calculations in lines 307-311. However, these do not (as yet) pose a problem. These will be left unchanged for now.
[10 Nov 2009 17:24] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/89982

2889 Chuck Bell	2009-11-10
      BUG#47698: backup.backup_logs fails under valgrind
      
      The backup_logs test contains a test statement designed to ensure 
      backup completes in less than 5 seconds. This goal is arbitrary
      and serves no purpose given the environment these tests execute.
      There could be significant delays in processing for virtual machines
      under heavy load thereby making any execution timing unpredictable.
      
      This patch removes the arbitrary timing test statement.
     @ mysql-test/collections/default.experimental
        Removed test from experimental list.
     @ mysql-test/suite/backup/r/backup_logs.result
        Corrected result file.
     @ mysql-test/suite/backup/t/backup_logs.test
        Removed arbitrary and inaccurate test statement.
[20 Feb 2010 9:19] Bugs System
Pushed into 6.0.14-alpha (revid:ingo.struewing@sun.com-20100218152520-s4v1ld76bif06eqn) (version source revid:ingo.struewing@sun.com-20100119103538-wtp5alpz4p2jayl5) (merge vers: 6.0.14-alpha) (pib:16)
[20 Feb 2010 19:07] Paul DuBois
Changes to test cases. No changelog entry needed.