Bug #39903 Tests that restart server fail sporadically on pushbuild
Submitted: 7 Oct 2008 9:18 Modified: 30 Jan 2009 17:39
Reporter: Sven Sandberg Email Updates:
Status: Closed Impact on me:
None 
Category:MySQL Server: Tests Severity:S7 (Test Cases)
Version:6.0-rpl OS:Any
Assigned to: Serge Kozlov CPU Architecture:Any
Tags: 6.0-rpl-green, maria-no-logging, pushbuild, sporadic, test failure
Triage: Triaged: D3 (Medium)

[7 Oct 2008 9:18] Sven Sandberg
Description:
maria-no-logging fails sporadically on pushbuild in 6.0-rpl. There are at least two common symptoms:

-------- FAILURE 1 --------
main.maria-no-logging                    [ fail ]

CURRENT_TEST: main.maria-no-logging
mysqltest: In included file "./include/maria_empty_logs.inc": At line 79: query 'use $default_db' failed: 2002: Can't connect to local MySQL server through socket '/dev/shm/pbtmp-n_mix-120/1/mysqld.1.sock' (2)

The result from queries just before the failure was:
set global maria_log_file_size=4294967295;
drop database if exists mysqltest;
create database mysqltest;
use mysqltest;
set global maria_checkpoint_interval=0;
create table t2 (a varchar(100)) engine=myisam;
insert into t2 select repeat('z',100);
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
* shut down mysqld, removed logs, restarted it

 - saving '/dev/shm/var-n_mix-120/1/log/main.maria-no-logging/' to '/dev/shm/var-n_mix-120/log/main.maria-no-logging/'

Retrying test, attempt(2/3)...

-------- END FAILURE 1 --------

-------- FAILURE 2 --------
main.maria-no-logging                    [ fail ]

CURRENT_TEST: main.maria-no-logging
mysqltest: In included file "./include/wait_until_connected_again.inc": At line 23: Server failed to restart

The result from queries just before the failure was:
< snip >
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
insert into t2 select * from t2;
* shut down mysqld, removed logs, restarted it
create table t1 (a varchar(100)) engine=maria transactional=1;
show create table t1;
Table	Create Table
t1	CREATE TABLE `t1` (
  `a` varchar(100) DEFAULT NULL
) ENGINE=MARIA DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 TRANSACTIONAL=1
show engine maria logs;
Type	Name	Status
MARIA	Size        16384 maria_log.00000001	unknown
insert into t1 values('a');
insert into t1 select * from t2;
show engine maria logs;
Type	Name	Status
MARIA	Size        24576 maria_log.00000001	unknown
* shut down mysqld, removed logs, restarted it

More results from queries before failure can be found in /tmp/var-ps_row-7/2/log/maria-no-logging.log

 - saving '/tmp/var-ps_row-7/2/log/main.maria-no-logging/' to '/tmp/var-ps_row-7/log/main.maria-no-logging/'

Retrying test, attempt(2/3)...

-------- END FAILURE 2 --------

How to repeat:
Failure 1: https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0-rpl&order=80 pb-valgrind/n_mix
xref: http://tinyurl.com/3wtg72

Failure 2: https://intranet.mysql.com/secure/pushbuild/showpush.pl?dir=bzr_mysql-6.0-rpl&order=80 sol10-amd64-a/ps_row
xref: http://tinyurl.com/4qcxey
[17 Oct 2008 8:48] Alexander Nozdrin
For the second failure -- see Bug#37810
[18 Oct 2008 16:43] Sven Sandberg
This is more general than maria-no-logging. It seems to happen for tests that have a server restart. See xref: http://tinyurl.com/6qudse

So it's probably a bug in the test framework.
[18 Oct 2008 16:47] Sven Sandberg
Clarification: my last comment refers to failure 2.

I think that BUG#37810 is a different issue (the failure is on a different line in wait_until_connected_again). Failure 2 happens only in 6.0-rpl, whereas BUG#37810 happens in other trees too.
[28 Oct 2008 17:47] Sven Sandberg
Failure 1 actually happens in other tests too, according to xref: http://tinyurl.com/6pby2g

It happens in:
  main.maria-autozerofill
  main.maria-no-logging
  main.maria-recovery
  main.maria-recovery2
  main.maria-recovery-bitmap
[24 Nov 2008 9:40] Lars Thalmann
I think it sounds like a good idea to check number of connections and
then wait for this number at restart.  For now perhaps use include
file and later when we have new test framework, this could be done as
a perl function.
[24 Nov 2008 22:42] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/59733

2738 Serge Kozlov	2008-11-25
      Bug#39903
      1. include/wait_for_status_var.inc: new primitive for waiting value of 
      a variable from SHOW STATUS.
      2. include/maria_empty_log.inc: added waiting of restoring all connections 
      to restarted server.
[25 Nov 2008 10:49] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/59768

2738 Serge Kozlov	2008-11-25
      Bug#39903: Sporadically some maria tests cannot use default connection 
      after server restart. It happens because the connections to server
      did not restored properly and a test should check them before using. 
      Suggested solution added checking number of connected threads on server 
      before restart and then waiting this number on restarted server.
      1. include/wait_for_status_var.inc: new primitive for waiting value of
      a variable from SHOW STATUS.
      2. include/maria_empty_log.inc: added waiting of restoring all connections
      to restarted server.
[25 Nov 2008 15:08] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/59807

2738 Serge Kozlov	2008-11-25
      Bug#39903: Sporadically some maria tests cannot use default connection
      after server restart. It happens because the connections to server
      did not restored properly and a test should check them before using.
      Suggested solution added checking number of connected threads on server
      before restart and then waiting this number on restarted server.
      1. include/wait_for_status_var.inc: new primitive for waiting value of
      a variable from SHOW STATUS.
      2. include/maria_empty_log.inc: added waiting of restoring all connections
      to restarted server.
[27 Nov 2008 11:56] Bugs System
A patch for this bug has been committed. After review, it may
be pushed to the relevant source trees for release in the next
version. You can access the patch from:

  http://lists.mysql.com/commits/60024

2739 Serge Kozlov	2008-11-27
      Bug#39903: Sporadically some maria tests cannot use default connection 
      after server restart. It happens because the connections to server
      did not restored properly and a test should check them before using. 
      Suggested solution added checking number of connected threads on server 
      before restart and then waiting this number on restarted server.
      1. include/wait_for_status_var.inc: new primitive for waiting value of
      a variable from SHOW STATUS.
      2. include/maria_empty_log.inc: added waiting of restoring all connections
      to restarted server.
[30 Jan 2009 13:30] Bugs System
Pushed into 6.0.10-alpha (revid:luis.soares@sun.com-20090129165607-wiskabxm948yx463) (version source revid:luis.soares@sun.com-20090129163120-e2ntks4wgpqde6zt) (merge vers: 6.0.10-alpha) (pib:6)
[30 Jan 2009 17:39] Paul Dubois
Test case changes. No changelog entry needed.