Bug #59063 rpl_migration_crash_safe fails on Windows
Submitted: 20 Dec 2010 16:27 Modified: 30 Jan 2011 18:11
Reporter: Alexander Nozdrin Email Updates:
Status: Closed Impact on me:
None 
Category:Tools Severity:S1 (Critical)
Version:5.6, 5.1, 5.5 OS:Any
Assigned to: Bjørn Munch CPU Architecture:Any
Tags: pb2, test failure
Triage: Triaged: D1 (Critical) / R1 (None/Negligible) / E1 (None/Negligible)

[20 Dec 2010 16:27] Alexander Nozdrin
Description:
rpl.rpl_migration_crash_safe w1 [ fail ]
        Test ended at 2010-12-20 12:50:41

CURRENT_TEST: rpl.rpl_migration_crash_safe
mysqltest: At line 78: 

The result from queries just before the failure was:
include/master-slave.inc
[connection master]
include/stop_slave.inc
CREATE TABLE test(id INTEGER NOT NULL PRIMARY KEY);
INSERT INTO test VALUES (1), (2), (3);
include/rpl_restart_server.inc [server_number=2 parameters: --relay-log-info-repository=TABLE --skip-slave-start]
"The mysql.slave_relay_log_info has information and this is not expected."

 - saving 'H:/pb2/test/sb_2-2695456-1292845372.28/mysql-5.6.1-m5-win-x86_64-test/mysql-test/var-n_mix/1/log/rpl.rpl_migration_crash_safe/' to 'H:/pb2/test/sb_2-2695456-1292845372.28/mysql-5.6.1-m5-win-x86_64-test/mysql-test/var-n_mix/log/rpl.rpl_migration_crash_safe/'

Retrying test rpl.rpl_migration_crash_safe, attempt(2/3)...

rpl.rpl_migration_crash_safe             w1 [ retry-fail ]

How to repeat:
Noticed in trunk-stage (a clone-off from trunk-bugfixing), probably
related to Sven's push of BUG#49978, BUG#59037.

Log: http://pb2.norway.sun.com/?action=archive_download&archive_id=2695767&pretty=please
[22 Dec 2010 21:30] Bugs System
Pushed into mysql-trunk 5.6.1 (revid:alexander.nozdrin@oracle.com-20101222212842-y0t3ibtd32wd9qaw) (version source revid:alexander.nozdrin@oracle.com-20101222212842-y0t3ibtd32wd9qaw) (merge vers: 5.6.1) (pib:24)
[14 Jan 2011 10:34] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/128712

3551 Sven Sandberg	2011-01-14
      BUG#59063: rpl_migration_crash_safe fails on Windows
      Backported the fix to 5.1.
      Problem: the auxiliary test files rpl_start_server.inc and rpl_stop_server.inc
      write a file that is later read by mtr. The bug was that the file was written
      with platform-dependent newline terminators, i.e., \r\n on windows, whereas mtr
      only understands \n.
      Fix: write the file so that it uses \n on all platforms.
     @ mysql-test/include/rpl_start_server.inc
        Force test to use \n instead of platform-dependent newline terminator.
     @ mysql-test/include/rpl_stop_server.inc
        Force test to use \n instead of platform-dependent newline terminator.
[14 Jan 2011 10:34] Bugs System
Pushed into mysql-5.1 5.1.56 (revid:sven.sandberg@oracle.com-20110114103312-ckj0gbf1zxf6bdsl) (version source revid:sven.sandberg@oracle.com-20110114103312-ckj0gbf1zxf6bdsl) (merge vers: 5.1.56) (pib:24)
[14 Jan 2011 10:37] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/128713

3244 Sven Sandberg	2011-01-14 [merge]
      merged BUG#59063 (backport from trunk) from 5.1 to 5.5
[14 Jan 2011 10:38] Bugs System
Pushed into mysql-5.5 5.5.10 (revid:sven.sandberg@oracle.com-20110114103653-uo8ocvbnbqi508cj) (version source revid:sven.sandberg@oracle.com-20110114103653-uo8ocvbnbqi508cj) (merge vers: 5.5.10) (pib:24)
[14 Jan 2011 10:41] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/128715

3506 Sven Sandberg	2011-01-14 [merge]
      Merged BUG#59063 from 5.5 to trunk.
      Since this was first fixed in trunk and then backported to 5.1 and
      merged up to 5.5, this was in effect a null merge.
[14 Jan 2011 10:41] Bugs System
Pushed into mysql-trunk 5.6.2 (revid:sven.sandberg@oracle.com-20110114104020-j0l7jpwzegs3d401) (version source revid:sven.sandberg@oracle.com-20110114104020-j0l7jpwzegs3d401) (merge vers: 5.6.2) (pib:24)
[14 Jan 2011 14:06] Bjørn Munch
Hmm, I think maybe this could have been solved with a fix to mtr. When using the restart:<options> syntax for expect file, mtr neglects to cut off the newline from the options part. It should have done so, and that I think would have avoided this problem.
[14 Jan 2011 14:17] Sven Sandberg
Björn, surely this could be fixed in mtr too. It would be better if mtr accepted any type of line ending. Feel free to implement that if you like...
[17 Jan 2011 12:37] Bjørn Munch
Are you sure this was the cause of the problem? AFAICS the test has only failed in one push on one widows build (out of 4). I've tried undoing your fix but the test does not fail.
[17 Jan 2011 14:22] Sven Sandberg
Bjørn, I don't know for sure if this was the problem. According to Alfranio who wrote the test, the failure indicates that the specified parameters were not given to mysqld. That failure would be explained if windows added line terminators that mtr did not understand. If this does not explain everything then there may be some other bug. I don't have time to dig too much in this now - if it shows up again, please re-open this bug. I think the pushed patch can't hurt in any case, so I'd say no action is needed for now.
[18 Jan 2011 11:30] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/129068

2973 Bjorn Munch	2011-01-18
      Bug #59063 rpl_migration_crash_safe fails on Windows
      Undoing the patch, it complicates the code but is not the solution
      
      I do not beleive newline mismatch could be the cause of this failure
      First, I cannot see how this could be a problem, mtr ignores the newline
        when reading the expect file, and the file is written and read on Windows.
      Second, if this really was the problem it should have been deterministic:
        either the newline is correctly interepreted or it is not.
[18 Jan 2011 11:36] Bjørn Munch
I suspect this could be an odd race condition similar to what had been observed a few times on Windows when running with --parallel. In a few cases, the .expect file had been deleted by mtr but the O/S claimed it still existed when mysqltest then tried to recreate it.

Perhaps mtr somehow reads the old version of the file.

Setting back to verified in case it happens again.
[19 Jan 2011 12:43] Bugs System
Pushed into mysql-trunk 5.6.2 (revid:bjorn.munch@oracle.com-20110119124232-d2o25k68680hjucy) (version source revid:bjorn.munch@oracle.com-20110119121742-s4rh30gg4y1s67fo) (merge vers: 5.6.2) (pib:24)
[19 Jan 2011 12:44] Bugs System
Pushed into mysql-5.5 5.5.10 (revid:bjorn.munch@oracle.com-20110119121645-58pj3ofc4dern1cy) (version source revid:bjorn.munch@oracle.com-20110119115617-i7gkn82c405te0o2) (merge vers: 5.5.10) (pib:24)
[19 Jan 2011 12:45] Bugs System
Pushed into mysql-5.1 5.1.56 (revid:bjorn.munch@oracle.com-20110119115537-st9m6zv8g18e29cf) (version source revid:bjorn.munch@oracle.com-20110119115537-st9m6zv8g18e29cf) (merge vers: 5.1.56) (pib:24)
[21 Jan 2011 11:57] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/129324

2975 Bjorn Munch	2011-01-21
      Bug #59063 rpl_migration_crash_safe fails on Windows
      Race condition may occur: mtr sees the .expect file but it's empty
      Fix: wait and try again if file is empty
      Also added verbose printout of extra restart options
[21 Jan 2011 12:02] Bjørn Munch
To reproduce on Windows, I modified the loop in check_expected_crash_and_restart() to 500 x 10ms instead of 50 x 100ms, then the test failed for me 1/5 of the time. It's probably very dependent on timing between mysqltest, mtr.pl and the O/S.
[21 Jan 2011 12:58] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/129331

2975 Bjorn Munch	2011-01-21
      Bug #59063 rpl_migration_crash_safe fails on Windows
      Race condition may occur: mtr sees the .expect file but it's empty
      Fix: wait and try again if file is empty
      Addendum: try again if line isn't 'wait' or 'restart'
      Also added verbose printout of extra restart options
[21 Jan 2011 14:24] Bjørn Munch
Pushed to -mtr branches
[25 Jan 2011 14:36] Bugs System
Pushed into mysql-trunk 5.6.2 (revid:bjorn.munch@oracle.com-20110125143535-lnouw5iqymn2g1uo) (version source revid:bjorn.munch@oracle.com-20110125141111-fl3ili63hgjz7l68) (merge vers: 5.6.2) (pib:24)
[25 Jan 2011 14:37] Bugs System
Pushed into mysql-5.5 5.5.10 (revid:bjorn.munch@oracle.com-20110125140853-4fs6w00hwho41aa2) (version source revid:bjorn.munch@oracle.com-20110125134645-f2dpdiycrx1k29qv) (merge vers: 5.5.10) (pib:24)
[25 Jan 2011 14:38] Bugs System
Pushed into mysql-5.1 5.1.56 (revid:bjorn.munch@oracle.com-20110125134426-l53nroj7m5tty5ga) (version source revid:bjorn.munch@oracle.com-20110125134426-l53nroj7m5tty5ga) (merge vers: 5.1.56) (pib:24)
[26 Jan 2011 0:27] Paul Dubois
Changes to test suite. No changelog entry needed.