Bug #43549 The backup_objects_dependency test is non-deterministic
Submitted: 10 Mar 2009 23:34 Modified: 25 Oct 2009 21:01
Reporter: Chuck Bell Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Backup Severity:S3 (Non-critical)
Version:6.0.11 OS:Any
Assigned to: Hema Sridharan CPU Architecture:Any
Tags: experimental, pb2, sporadic, test failure

[10 Mar 2009 23:34] Chuck Bell
Description:
The backup_objects_dependency test is non-deterministic.

The test makes use of an event ob3.ev which updates a table that is later queried for values. Since the firing of events is undependable when using TIMESTAMP (slower systems or systems under heavy load -- like PB -- can delay the firing of the event), the query against the table that the event updates is also unreliable.

You can see random failures in pushbuild due to content mismatch:

CURRENT_TEST: backup.backup_objects_dependency
--- /data0/pushbuild/pb/bzr_mysql-6.0-backup/200/mysql-6.0.11-alpha-pb200/mysql-test/suite/backup/r/backup_objects_dependency.result	2009-03-10 23:49:23.000000000 +0300
+++ /data0/pushbuild/pb/bzr_mysql-6.0-backup/200/mysql-6.0.11-alpha-pb200/mysql-test/suite/backup/r/backup_objects_dependency.reject	2009-03-11 01:42:08.000000000 +0300
@@ -538,7 +538,6 @@
 ) ENGINE=MyISAM DEFAULT CHARSET=latin1
 SELECT * FROM ob3.log;
 pos	msg
-1	user1 created
 CALL show_objects('ob1');
 TABLE_NAME	TABLE_TYPE
 t11	BASE TABLE

and

CURRENT_TEST: backup.backup_objects_dependency
--- /data0/pushbuild/pb/bzr_mysql-6.0-backup/200/mysql-6.0.11-alpha-pb200/mysql-test/suite/backup/r/backup_objects_dependency.result	2009-03-10 23:49:23.000000000 +0300
+++ /data0/pushbuild/pb/bzr_mysql-6.0-backup/200/mysql-6.0.11-alpha-pb200/mysql-test/suite/backup/r/backup_objects_dependency.reject	2009-03-11 01:41:53.000000000 +0300
@@ -851,7 +851,6 @@
 SELECT * FROM ob3.log;
 pos	msg
 1	user1 created
-2	user2 created
 
 CREATE DEPENDENT TESTS
 ======================

How to repeat:
Examine the pushbuild runs for mysql-6.0-backup and mysql-6.0-backup-merge.

Suggested fix:
Add synchronization points to the test to wait until the table(s) are updated.
[11 Mar 2009 1:08] Chuck Bell
The use of synchronization points will not work because the event is being executed in a separate thread and therefore the synchronization code will not fire correctly.

Recommend rewriting the test to ensure the event is fired before selects are made against the tables the event updates.

There are three sections of the test that have been disabled and marked by this bug report.
[11 Mar 2009 1:12] Chuck Bell
CORRECTION: There are four places in the test that need attention.
[11 Mar 2009 1:15] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/68823

2797 Chuck Bell	2009-03-10
      Disabled portions of test pending patch for BUG#43549.
      modified:
        mysql-test/suite/backup/r/backup_objects_dependency.result
        mysql-test/suite/backup/t/backup_objects_dependency.test
[11 Mar 2009 8:16] Jørgen Løland
Resetting status to Verified - the pushed patch is a temporary solution.
[26 Mar 2009 12:34] Bugs System
Pushed into 6.0.11-alpha (revid:alik@sun.com-20090326121822-pt84kzxxayzho4mn) (version source revid:charles.bell@sun.com-20090311011833-m6wegzcn4j35xnn2) (merge vers: 6.0.11-alpha) (pib:6)
[14 Apr 2009 15:45] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/72028

2794 Hema Sridharan	2009-04-14
      Bug#43549. The backup_objects_dependency test will now show deterministic results 
      by using wait conditions when an event is fired
      modified:
        mysql-test/suite/backup/r/backup_objects_dependency.result
        mysql-test/suite/backup/t/backup_objects_dependency.test
[29 Apr 2009 2:56] Hema Sridharan
This bug is waiting for fix of Bug#44533
[18 Aug 2009 20:32] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/81021

2861 Hema Sridharan	2009-08-18
      BUG#43549 (The test backup_objects_dependency is 
      non-deterministic).
      The fix will use wait condition when events are firing and 
      produce deterministic results.
      modified:
        mysql-test/suite/backup/r/backup_objects_dependency.result
        mysql-test/suite/backup/t/backup_objects_dependency.test
[19 Aug 2009 12:47] Chuck Bell
While we need only one reviewer and Jorgen has taken that responsibility, I must add a comment here.

I think the test as written has a level of risk that it will fail. The reason I say that is the test is using a procedure to create and execute an event which I feel is risky because the procedure runs in another thread. This extra step is not really needed because the scheduler can be turned on and off and the event altered in line. 

I have recommended a change to the test as an external observer but have not required the test to be changed. If it proves that the test eventually fails in pushbuild, then the test must be changed to remove the use of the stored procedure.
[19 Aug 2009 12:47] Chuck Bell
Setting to verified because patch was rejected by first reviewer.
[19 Aug 2009 17:13] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/81101

2861 Hema Sridharan	2009-08-19
      BUG#43549 (backup_objects_dependency.test is non deterministic)
      The fix will use wait condition when events are fired and produce
      deterministic results.
      modified:
        mysql-test/suite/backup/r/backup_objects_dependency.result
        mysql-test/suite/backup/t/backup_objects_dependency.test
[20 Aug 2009 18:27] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/81201

2862 Hema Sridharan	2009-08-20
      BUG#43549 (backup_objects_dependency test is non-deterministic)
      The fix will use wait conditon when events are firing and produce
      deterministic results.
      modified:
        mysql-test/suite/backup/r/backup_objects_dependency.result
        mysql-test/suite/backup/t/backup_objects_dependency.test
[20 Aug 2009 18:31] Hema Sridharan
Pushed the patch as Jorgen as approved the new patch. Marking the reviewer box for Jorgen.
[20 Aug 2009 18:31] Hema Sridharan
There is no need of 2nd reviewer as per Chuck.
[14 Sep 2009 10:55] Ingo Strüwing
I am sorry that I need to reopen this. It still fails often. This time the most frequent problem is a timeout in the wait_condition. Please see the cross-reference page: http://pb2.norway.sun.com/web.py?template=mysql_show_test_failure&test_failure_id=2314231&...
[15 Sep 2009 23:48] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/83392

2868 Hema Sridharan	2009-09-16
      BUG#43549 (backup_objects_dependency tests results are non 
      deterministic) The patch will remove the wait condition and 
      remove the check wheather event is fired or not as it 
      produces non deterministic results.
      removed:
        mysql-test/suite/backup/t/backup_row_size.test
      modified:
        mysql-test/suite/backup/r/backup_objects_dependency.result
        mysql-test/suite/backup/t/backup_objects_dependency.test
[17 Sep 2009 21:04] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/83656

2870 Hema Sridharan	2009-09-17
      BUG#43549 (backup_objects_dependency test is non
      deterministic). The new patch for this bug will remove all
      the dependencies of events and check for events execution. 
      modified:
        mysql-test/suite/backup/r/backup_objects_dependency.result
        mysql-test/suite/backup/t/backup_objects_dependency.test
[25 Oct 2009 13:38] Bugs System
Pushed into 6.0.14-alpha (revid:alik@sun.com-20091025133616-ca4inerav4vpdnaz) (version source revid:ingo.struewing@sun.com-20090908195642-dtq0vxjcjk6e11w4) (merge vers: 5.4.4-alpha) (pib:13)
[25 Oct 2009 21:01] Paul DuBois
Test case changes. No changelog entry needed.