Bug #41173 rpl_packet fails sporadically on pushbuild: query 'DROP TABLE t1' failed
Submitted: 2 Dec 2008 14:14 Modified: 21 Jan 2009 16:20
Reporter: Sven Sandberg Email Updates:
Status: Closed Impact on me:
None 
Category:Tests: Replication Severity:S7 (Test Cases)
Version:6.0-rpl, 5.1-rpl OS:Any
Assigned to: Andrei Elkin CPU Architecture:Any
Tags: 6.0-rpl-green, pushbuild, rpl_packet, sporadic, test failure
Triage: Triaged: D3 (Medium)

[2 Dec 2008 14:14] Sven Sandberg
Description:
sporadic pushbuild failure in 6.0-rpl, debx86-b/ps_row:

rpl.rpl_packet                           [ fail ]

CURRENT_TEST: rpl.rpl_packet
mysqltest: At line 89: query 'DROP TABLE t1' failed: 1051: Unknown table 't1'

The result from queries just before the failure was:
< snip >
count(*)
1
SHOW STATUS LIKE 'Slave_running';
Variable_name	Value
Slave_running	ON
select * from information_schema.session_status where variable_name= 'SLAVE_RUNNING';
VARIABLE_NAME	VARIABLE_VALUE
SLAVE_RUNNING	ON
drop database DB_NAME_OF_MAX_LENGTH_AKA_NAME_LEN_64_BYTES_____________________;
SET @@global.max_allowed_packet=4096;
SET @@global.net_buffer_length=4096;
include/stop_slave.inc
include/start_slave.inc
CREATE TABLe `t1` (`f1` LONGTEXT) ENGINE=MyISAM;
INSERT INTO `t1`(`f1`) VALUES ('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa2048');
Slave_IO_Running = No (expect No)
==== clean up ====
DROP TABLE t1;
SET @@global.max_allowed_packet= 1024;
SET @@global.net_buffer_length= 1024;

More results from queries before failure can be found in /dev/shm/var-ps_row-101/1/log/rpl_packet.log

 - saving '/dev/shm/var-ps_row-101/1/log/rpl.rpl_packet/' to '/dev/shm/var-ps_row-101/log/rpl.rpl_packet/'

Retrying test, attempt(2/3)...

How to repeat:
https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0-rpl&order=99 debx86-b/ps_row
xref: http://tinyurl.com/66jk4g
[2 Dec 2008 14:18] Sven Sandberg
Other failure in the same test: BUG#33420
[2 Dec 2008 15:00] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/60399

2703 Andrei Elkin	2008-12-02
      bug#41173 rpl_packet fails sporadically on pushbuild: query 'DROP TABLE t1' failed 
      
      The problem is most probably is missing synchronization in mtr with the slave io thread state.
      The io thread is not guaranteed to be connected by the end of `start slave'.
      
      Fixed with deploying wait_for_slave_to_start.inc macro invocation.
[2 Dec 2008 15:22] Andrei Elkin
The former patch is useless. I did not notice there has been the synchronization in the test that I was trying to make.
[2 Dec 2008 18:40] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/60417

2742 Andrei Elkin	2008-12-02
      Bug #41173  rpl_packet fails sporadically on pushbuild: query 'DROP TABLE t1' failed
      
      This one is not a fix but a temporary addon to the test to print extra debug.
      The test is also got narrowed to run in the row binlog format for time the bug is being fixed.
      
      Todo: remove the debug prints and the format restriction within the real bug fix.
[2 Dec 2008 22:17] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/60432

2743 Andrei Elkin	2008-12-03
      Bug #41173 rpl_packet fails sporadically on pushbuild: query 'DROP TABLE t1' failed
      
      The issue appeared to be a race between the SQL thread executing CREATE table t1
      and the IO thread that is expected to stop at the consequent big size event.
      
      The two events need serialization which is implemented.
[4 Dec 2008 14:23] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/60612

2734 Andrei Elkin	2008-12-04
      Bug #41173 rpl_packet fails sporadically on pushbuild: query 'DROP TABLE t1' failed
      Bug #33420 Test 'rpl_packet' fails randomly with changed "Exec_Master_Log_Pos"
      
      The both issues appeared to be a race between the SQL thread executing CREATE table t1
      and the IO thread that is expected to stop at the consequent big size event.
      
      The two events need serialization which is implemented.
      The early bug required back-porting a part fixes for bug#38350 exclusively for 5.0 version.
[9 Dec 2008 17:46] Andrei Elkin
pushed to bugteam trees, starting from 5.0.
[15 Jan 2009 6:33] Bugs System
Pushed into 5.1.31 (revid:joro@sun.com-20090115053147-tx1oapthnzgvs1ro) (version source revid:azundris@mysql.com-20081230114838-cn52tu180wcrvh0h) (merge vers: 5.1.31) (pib:6)
[16 Jan 2009 3:49] Paul Dubois
Test case changes. No changelog entry needed.

Setting report to NDI pending push into 6.0.x.
[19 Jan 2009 11:30] Bugs System
Pushed into 5.1.31-ndb-6.2.17 (revid:tomas.ulin@sun.com-20090119095303-uwwvxiibtr38djii) (version source revid:tomas.ulin@sun.com-20090115073240-1wanl85vlvw2she1) (merge vers: 5.1.31-ndb-6.2.17) (pib:6)
[19 Jan 2009 13:07] Bugs System
Pushed into 5.1.31-ndb-6.3.21 (revid:tomas.ulin@sun.com-20090119104956-guxz190n2kh31fxl) (version source revid:tomas.ulin@sun.com-20090119104956-guxz190n2kh31fxl) (merge vers: 5.1.31-ndb-6.3.21) (pib:6)
[19 Jan 2009 15:45] Jon Stephens
Setting status back to NDI pending merge to 6.0 tree.
[19 Jan 2009 16:13] Bugs System
Pushed into 5.1.31-ndb-6.4.1 (revid:tomas.ulin@sun.com-20090119144033-4aylstx5czzz88i5) (version source revid:tomas.ulin@sun.com-20090119144033-4aylstx5czzz88i5) (merge vers: 5.1.31-ndb-6.4.1) (pib:6)
[20 Jan 2009 18:59] Bugs System
Pushed into 6.0.10-alpha (revid:joro@sun.com-20090119171328-2hemf2ndc1dxl0et) (version source revid:azundris@mysql.com-20081230114916-c290n83z25wkt6e4) (merge vers: 6.0.9-alpha) (pib:6)
[21 Jan 2009 16:20] Paul Dubois
Test case changes. No changelog entry needed.
[30 Jan 2009 13:31] Bugs System
Pushed into 6.0.10-alpha (revid:luis.soares@sun.com-20090129165607-wiskabxm948yx463) (version source revid:luis.soares@sun.com-20090129163120-e2ntks4wgpqde6zt) (merge vers: 6.0.10-alpha) (pib:6)